10 stories

Using a HELOC as an investment strategy: not as taboo as you might think

1 Comment

Q. I wish to leverage my home equity line of credit (HELOC) to invest in dividend-paying investments. How would you advise I approach this? Is this an effective tax savings tool? Is there any financial institution or products you would advise?

A. You know, Martha, in some circles leveraging—or borrowing to invest—is a taboo subject. I find that funny, because there is much less controversy when people borrow to:

  • Buy a car, which depreciates in value;
  • Buy a house, which normally appreciates, but can decline;
  • Take a vacation as a lifestyle investment.

So why is there controversy around borrowing to invest? Probably a lack of understanding, coupled with the fact that when leveraging goes bad, it’s not good.

Let’s talk about leverage. If you borrow $100,000 at 5%, what rate of return would you have to earn on your investments to break even?  Would you guess 5%?

Most people would agree with that answer; it sounds logical, right? I mean, if you borrow $100,000 at 5% and paid $5,000 in interest costs then that would mean you would have to make $5,000 on your $100,000 investment to break even, which is 5%. Got it? Good.

But that answer is wrong!

The break-even return on investments is lower than the borrowing cost when you take into account:

  1. Simple vs. compound interest over time;
  2. The interest tax deduction and;
  3. The tax deferral on the investments.

Still skeptical? Consider a free trial of Talbot Stevens’ leverage software to see for yourself. (Here is a demonstration of the software.)

When you make annual interest payments on a loan, this is considered simple interest. For example, in the first year, you would pay $5,000 in interest charges on a $100,000 loan at 5%. In the second year, you’d have the same interest charge because the loan is still $100,000. If you plotted your total interest payments over time on a graph you would see a straight line sloping up toward the right.

Contrast that with a $100,000 investment earning 5%. After the first year, you’ll have $105,000. In the second year you’ll have $105,000 plus 5%, which brings the running total to $110,250—and up it goes each year. This is compound interest, which would produce a graph that curves upward toward the right.

To summarize that point: when you make annual interest payments on a loan, simple interest applies, whereas investments compound. The longer you hold the investments, the greater the compound effect and the lower the return needs to be on the investments to break even.

Now, let’s consider the tax deduction. When you borrow money to invest, the interest cost is considered a carrying charge on your tax return, which creates a tax deduction no different than an RRSP contribution. Looking at this in concrete terms, if your marginal tax rate is 30%, your after-tax cost of borrowing is 3.5%. Now, as with any strategy, I’d recommend you take the tax savings and invest it or pay down non-tax deductible debt.  The benefit of investing the savings earned through your tax deduction is not just that you potentially build more wealth, but that it becomes easier to measure the success or failure of the leveraging strategy. Many people look only at the value of the leveraged investments and forget about, or discount, the value of the tax deduction. You need to account for the value of the tax deduction to get a full picture, and reinvesting it is a simple way to do that.

The final point is the tax efficiency of your investments. The less tax you pay on your investments as they grow, the more money you have invested, and the more the return compounds over time. With that in mind, do dividend-paying investments make sense? What’s your reason for selecting dividend-paying investments? Is it because you feel those investments are safe and provide good returns? That’s an OK reason, but if you could find investments with a similar level of risk and rate of return which paid less in distributions/income, then you may be better off from a tax perspective.

Were you thinking of using the dividends to pay some of the interest charges? You can do that but I wouldn’t advise it. Don’t do leverage unless you can easily make the interest payments out of your regular cash flow.

Here are a few things to consider:

  1. Use a separate HELOC for your source of investing funds. This will make your taxes easier.
  2. Hold a separate account just for the leveraged investments; again, this will help with tax accounting.
  3. While you’ll want to invest according to your specific profile, one possible strategy is to buy broad market index investments.
  4. Don’t use systematic withdrawals from the investment to make interest payments.
  5. Plan to invest for 10 years or more.
  6. Remember, leveraging magnifies returns up and down. If your $100,000 goes to $80,000 and you have to sell, you will still owe the bank  that “lost” $20,000 (the difference between the $100,000 you borrowed to invest, and the $80,000 value of your investment when you sold).

Finally, when it comes to leverage don’t think just about investment accumulation but also think about how you can use the interest tax deduction. Here are a few quick thoughts:

  1. Will it reduce your income so you can get more of the Canada Child Tax Benefit, GIS, OAS, and the Age Credit?
  2. Will the tax deduction offset the tax owed on RRIF or INVESTCO withdrawals?
  3. Can you use the tax deductions to pay off your mortgage faster and accumulate investments i.e. the Smith Manoeuvre.
  4. Interest on the interest that is tax deductible is also tax deductible so consider paying off non-tax-deductible debt before making interest payments on your leveraged loan.

I hope I’ve given you some things to think about Martha and I haven’t made it look like everyone should run out and start leveraging. If you have the cash flow, borrow within your means, stick to a broad market investment, and have a long time horizon, you’ll greatly improve your odds for success.

Allan Norman is a certified financial planner and Chartered Investment Manager with Atlantis Financial Inc. in Barrie, Ont. He can be reached at alnorman@atlantisfinancial.ca

This commentary is provided as a general source of information and is intended for Canadian residents only. Allan offers financial planning services through Atlantis Financial Inc.













The post Using a HELOC as an investment strategy: not as taboo as you might think appeared first on MoneySense.

Read the whole story
1897 days ago

Share this story

Two simple and cheap portfolios for responsible investors

1 Share

When I started my Sustainable Economist blog in 2012, there were only a small handful of socially responsible and green Exchange Traded Funds (ETFs). With so few options available, it was challenging to create model portfolios that were properly diversified. My, how times have changed! It feels like new sustainability ETFs are coming out all the time, and now my challenge is to stay on top of them all.

For those new to this type of investing, socially responsible investors encourage corporate practices that promote environmental stewardship, consumer protection, human rights, and diversity.  Some avoid businesses involved in tobacco, alcohol, gambling, pornography, fossil fuel production, and guns.

This year, I’m introducing two new model portfolios for those looking for an investment strategy that takes into account both the financial strategy and the social/environmental good to bring out a positive change in our world. I outline them below.

The Vanguard Cheap & Easy ESG Portfolio

Asset Class Allocation ETF Name Ticker MER Currency
US Equity 30% Vanguard ESG U.S. Stock ETF ESGV 0.12 USD
Int’l Equity 30% Vanguard ESG International Stock ETF VSGX 0.15 USD
Gov’t Bonds 40% Vanguard Canadian Government Bond Index ETF VGV 0.28 CAD

I’ve long admired Vanguard’s unique ownership structure and their mission to keep fees low for investors. Imagine my excitement when they launched their new ETFs with a socially responsible mandate. I even got to write about their pros and cons for MoneySense at that time.

This portfolio is by no means perfectly sustainable, but it is hyper diversified and very cheap with a total Management Expense Ratio (MER) of just 0.19%. It is a step in the right direction for people who want to dip their toes into sustainable investment strategies. Curiously, Vanguard’s Canadian Government Bond ETF (VGV) is not the cheapest Canadian government bond ETF on the market so investors can lower their total MER to 0.15% by swapping it out for less expensive options offered by BMO and iShares.

The iShares Impact Portfolio

Asset Class Allocation ETF Name Ticker MER Currency
Canadian Equity 15% iShares Jantzi Social Index ETF XEN 0.55 CAD
US Equity 15% iShares MSCI KLD 400 Social ETF DSI 0.25 USD
Int’l Developed Equity 10% iShares ESG MSCI EAFE ETF ESGD 0.2 USD
Int’l Emerging Equity 5% iShares ESG MSCI EM ETF ESGE 0.25 USD
Impact Equity 15% iShares MSCI Global Impact ETF SDG 0.49 USD
Canadian Bonds 30% iShares 1-10 Year Laddered Government Bond Index ETF CLG 0.17 CAD
Impact Bonds 10% Oikocredit GIC / Term Deposit 0 CAD

iShares has long been a leader in socially responsible ETFs, having created the Canadian flagship Jantzi Social Index ETF (XEN) back in 2007. More recently, they’ve expanded their lineup of socially responsible ETFs. I expect more to launch in the coming years as Blackrock CEO Larry Fink sees sustainable investing as a massive trend, and predicted in a recent interview that it will become “a core component of how everyone invests in the future… We are only at the early stages”.

Slightly more expensive with a total MER of 0.28%, this approach is for investors who want to go beyond ‘doing less evil’ by carving out part of their portfolio for investments that are ‘doing more good’. It includes the iShares MSCI Global Impact ETF (SDG), which invests in companies that provide goods and services aligned with the United Nations Sustainable Development Goals. It also includes the Oikocredit GIC/Term Deposit, an amazing investment that generates a stable financial return while providing micro-loans to entrepreneurs and co-ops in emerging economies. The Oikocredit GIC is available to Ontario investors through Kindred Credit Union and the Oikocredit Term Deposit is available to B.C. residents through Vancity.

Tim Nash is Founder of Good Investing, an investment coaching firm that empowers everyday people to manage their own money according to their values.


The post Two simple and cheap portfolios for responsible investors appeared first on MoneySense.

Read the whole story
2016 days ago
Share this story

Everything an investor needs in a single ETF

1 Share

Vanguard Investments Canada Inc. has announced the listing of three new low-cost Asset Allocation ETFs that give investors one-stop shopping to the firm’s globally diversified strategies. This is a significant move that not only creates a smart, low-fee, all-in-one portfolio but will also act as a challenge to the rise of so-called robo advisors.

In essence, the middle (Balanced) of the three Asset Allocation ETFs is the equivalent of the global balanced fund, which I’ve argued in the past should — in theory anyway — be the only investment fund you need. Similarly, while Vanguard’s ETFs are invariably components of the robo-adviser services out there (along with BlackRock iShares), any of these three new ETFs could serve as a one-size-fits all alternative to them. It also compares with Franklin Templeton’s Quotential.

The difference is that at 22 basis points and change, the three Vanguard products are quite a bit less costly: less than half what many robo services charge, which is typically about 50 basis points (half a per cent) plus the underlying ETF fees.

Both investors and advisors are asking for “simple yet sophisticated single-ticket investment solutions that provide well-diversified global equity and bond exposure within a low-cost ETF structure,” says Atul Tiwari, managing director for Vanguard Canada. The new ETFs offer investors three different risk profiles and regular rebalancing.

In effect, each ETF is a fund of funds although Vanguard describes them as having an “ETF of ETFs structure.” Each holds seven existing core Vanguard index ETFs (which I list in the postscript below). Each new ETF of ETFs has a management Fee of 0.22%. Vanguard says that when one of its ETFs invests in underlying Vanguard funds, “there shall be no duplication of management fees.” Spokesman Matthew Gierasimczuk said “There are no duplicate fees beyond the 0.22 management fee, other than a basis point or two for operating expense and the trading fee for buying or selling the ETF.”

RELATED: How to declutter your portfolio

The three asset allocation ETFs cover the normal range from Conservative to Balanced to Growth, as reflected in the product names. Equity weights range from 40% for the Conservative offering, to 60% for the Balanced and 80% for the Growth.

Certainly for do-it-yourself investors who no longer want to be constantly watching the markets and the disparate movements of individual stocks, any of these three new ETFs could be a good substitute and means for getting your life back. Younger people should pick the Growth version with 80% stocks, cautious mid-career people could favour the balanced 60/40% stocks version and older investors and retirees could choose the conservative version with only 40% stocks.

Since it’s arguable even retirees need that high a proportion of stocks to hedge against the possible future ravages of inflation, it too makes sense, although 40% stocks may be a tad high for older retirees. Alternatively, they could put half in the Conservative ETF and another half in ladders of two-year GICs, taking the combined equities down to a very conservative 20%.

It’s not clear to me whether fee-based advisors will flock to these products, although here too it would free them up to do true financial planning, deal with taxes, estate planning and other minutiae. The really good ones might well gravitate to it; the asset gatherers not so much.

Ideal for TFSAs

I also see this as a core holding in Tax-free Savings Accounts (TFSAs), since the $5,500 current annual contribution limit constitutes a relatively small “ticket” and these give you all the world and all asset classes in a single punch of the ticket. The Growth version (VGRO) certainly has healthy exposure to both Canadian and global securities, both stocks and bonds.

According to the fact sheet, it is 30.1% in the Vanguard U.S. Total Market Index ETF, 24% the Vanguard FTSE Canada All Cap Index ETF, 20% in the Vanguard FTSE Developed All Cap ex North America Index ETF, and 5.9% in the Vanguard FTSE Emerging Markets All Cap Index ETF. That’s exposure to the whole world’s stocks. The 20% fixed income comes from 11.7% in the Vanguard Canadian Aggregate Bond Index ETF, 4.7% in the Vanguard Global ex-US Aggregate Bond Index EFF (CAD Hedged), and 3.6% in the Vanguard U.S. Aggregate Bond Index ETF (CAD Hedged). All rebalanced regularly!

RELATED: How ETF investors sabotage themselves

Because TFSAs can still be added to into advanced old age — my 101-year old friend Meta still contributes to hers! — I’d lean to going with the Growth or Balanced versions, and view the Conservative one as more appropriate for RRSPs and RRIFs, particularly the latter once forced annual minimum withdrawals commence (and therefore need to generate cash).

Finally, for couples where one spouse is the “finance” person and the other disinterested, this kind of product seems ideal for older do-it-yourself investors who are beginning to worry that dementia and related ills might impair their cognitive skills for investing.

In the case of a financially literate wife and a financially uninterested husband, if the wife were concerned about her dying first and wishing to put the collective portfolio on autopilot, the Balanced or Conservative version might also do the trick, short of just handing the whole lot over to a professional money manager.

Here are the 3 ETFs and their ticker symbols on the TSX:

  • Vanguard Conservative ETF Portfolio (VCNS) seeks to provide a combination of income and moderate long-term capital growth by investing in equity and fixed income securities with a strategic allocation of 40% equities and 60% fixed income.
  • Vanguard Balanced ETF Portfolio (VBAL) will provide long-term capital growth with a moderate level of income split 60% equities to 40% fixed income.
  • Vanguard Growth ETF Portfolio (VGRO) provides long-term capital growth by investing in equity and fixed income securities with 80% equities and 20% fixed income.

In a press release, Vanguard Canada head of product Tim Huver said the ETFs offer “a simplified and scalable solution for financial advisors, and a one-stop globally-diversified and transparent option for investors … Investors can rely on Vanguard’s global investment experts to continuously assess their portfolio’s exposure and rebalance it back to its intended risk level.”

With the three new ETFs, Vanguard Canada now offers 36 ETFs, with C$14 billion in assets under management. Vanguard Investments Canada Inc. is a wholly owned indirect subsidiary of The Vanguard Group, Inc.

A longer version of this story ran at the author’s Financial Independence Hub

Jonathan Chevreau is founder of the Financial Independence Hub and co-author of Victory Lap Retirement. He can be reached at jonathan@findependencehub.com.    


The post Everything an investor needs in a single ETF appeared first on MoneySense.

Read the whole story
2359 days ago
Share this story

The End of the Rainbow

1 Comment and 14 Shares
The retina is the exposed surface of the brain, so if you think about a pot of gold while looking at a rainbow, then there's one at BOTH ends.
Read the whole story
2374 days ago
So, if I put on a leprechaun costume, there _will_ be one at the end of the rainbow.
2355 days ago
nailed it
Share this story

leaving the office after a successful deployment

1 Share
/* by MollDenn */
Read the whole story
3665 days ago
Share this story

Speeding Up Your Engineering Org, Part I: Beyond the Cost Center Mentality

1 Share

It is a truth universally acknowledged, that engineering orgs—like greyhounds, sports cars, and wide receivers—slow down as they age.

Odds are good that you have experienced this phenomenon personally at some point in your engineering career. The slowdown was gradual, frustrating, and oddly stubborn. It survived: numerous rounds of hiring; a spate of offsites where inspiring speakers harangued everyone to “cut through the crap” and just “get shit done”; a blood-spattered re-org or two; and even a few ground-up rewrites that utterly failed to deliver on their promised boost in velocity.

If you’re now involved with engineering leadership in some capacity, you may well have accepted the slowdown as a sad universal truth. Accordingly, you may have shifted your efforts from the impossible task of making the org go faster to the thankless but crucial job of jealously guarding how engineers spend their time—because as it takes longer and longer to get even simple features out the door, those engineering hours become increasingly precious.

If all this sounds familiar, I have good news and bad news for you.

The good news: it isn’t actually a law of nature that engineering orgs have to slow down as they mature and grow. With active, contravening investment, it’s possible to maintain and even gain speed.

But,” you protest, “I’ve made investments, remember? I’ve hired! I’ve brought in speakers! I’ve re-orged and re-factored and tried out every flavor of agile there is, and still we go slower and slower!”

Yes, which brings us to the bad news: that slowdown is a far bigger deal than you might have realized, and way more harmful to the bottom line of your business than you might imagine. Oh, and that jealous guarding of engineer hours for features? It’s only making things worse.

In this article I’m going to consider the speed of an engineering org as an economic question—not a moral question, or a question of technology choices, or a question of people “hustling” and “powering through” the obstacles they find in their path. I believe that a good percentage of engineering and business leaders economically model their engineering org—consciously or unconsciously—as a “cost center,” where every engineer hour not spent on features must translate to (at least) one engineer hour saved, and I believe that this economic model makes it extremely difficult to identify and justify the investments that could actually speed that org up. I’ll propose an alternate economic model of an engineering org—one in which speed to delivery, rather than number of engineer hours paid, is the dominant economic factor—and in which considerable, sustained investment in that speed can reap massive economic returns.

But let’s get a little more concrete with this—let’s look at an example of the kinds of decisions that face engineering orgs and their leaders every day, and just how easy it is to slip into the “cost center” mentality when attempting to juggle them.

A Tale of Two Engineers

Say you’re an engineering manager at Company X, and one morning you arrive at work to find two of your best engineers waiting outside your office. You haven’t even opened your door before they start in on you.

Look,” says Cindy, the first engineer, “I know that the CEO is breathing down our neck to finish the new Facebook for Cats integration, but we’ve got to clear some time to work on automating database migrations. I’m the only one who knows enough to apply them to the prod DB, and I’m getting tired of spending half an hour every morning rolling out everyone else’s changes. So can we push a feature or two back and squeeze that in?”

Forget the migrations,” says Scott, the second engineer, “we need to talk about the Frobulator Service. Two years ago we agreed to hack it up quickly in PHP, but product promised us—PROMISED—that we would have time to go back and clean it up. Yesterday I happened to be back in that code while I was updating the copyright years in our headers, and it’s even worse than I remembered. We need to rewrite it in Scala so it’s more modern, performant, and easier to maintain. Can you tell product we’re calling in that promise, please, and I’ll get started?”

First off: everything your engineers have said is true. Cindy really is spending a half hour every morning dealing with database migrations; the source for the Frobulator Service really does look like a plate of partially digested capellini; product really did promise time to clean that mess up; and of course there really is a long and growing backlog of features for the upcoming Facebook for Cats integration, each of them (according to the CEO and product) absolutely essential and destined to become a customer favorite.

Furthermore, you’ve been around long enough to know that there won’t be any “calm periods” when there’s time for your engineers to scratch these other itches—after the Facebook for Cats integration goes out, you’ll be right on to integrating with Twitter for Dogs, or LinkedIn for Ferrets. So on this fine morning someone has to make a real and uncomfortable decision: either tell Cindy and Scott to stop complaining and get back to feature work, or let product and the CEO know that you’re going to spend some engineering hours on something other than features. And today that someone is you.

Pop quiz, hot shot: what do you do?


A Simple, Responsible, and Totally Wrong Approach

If you’re a mature, business-focused engineering leader, you might grab some coffee, sit Cindy and Scott down, and tell them something like this:

Cindy, I’m sorry to hear that you’re getting bored doing so much production DB work, but realistically it would take you at least 40 hours of work to write, test, and deploy a migration utility, right? So if you’re spending a half hour a day on migrations, it would be 80 working days before we saw a return on our investment—that’s like 4 months, and that’s just too long for me to sanction—precisely because you’re such a valuable member of the team, and I can’t spare so much of your time right now away from our feature backlog. We can touch base if the migration workload increases too much, OK? Until then, I have to ask you to put your head down and be a team player.

Scott, you’re absolutely right, product did promise that we could spend time cleaning up the Frobulator Service, and I’m sure they were acting in good faith, but none of us could have possibly known at the time how our product was going to take off—we’ve got customers practically beating down our door for new features, and they’re not going to see any difference whether the Frobulator Service is written in crappy PHP or transcendent Scala.

Both of you are great engineers with bright futures, and if those futures include engineering management, then part of your job will be to understand that engineering’s job is to produce effects that are visible to customers. So if we burn hours on projects that aren’t customer visible—projects that are by engineers, for engineers—we need to be able to show directly how those hours will pay for themselves in saved engineering hours in pretty short order.”

This approach feels rational, responsible, and easy to apply, right? There’s only one small problem: by slipping into the “cost center” mentality, where engineering hours must only be spent on features or a greater savings in engineering hours, you’ve actually just slowed your engineering org down further, and cost your company real (though largely invisible) money in the process. How did this happen without our even noticing, while we thought we were being so responsible?

Engineer Hours” vs. Latency—Where the “Cost Center” Gets it Wrong

The cost center model of engineering, to which our hypothetical engineering leader has just retreated, is basically this: an engineering org is a furnace which burns money, in the form of compensated engineer hours, and produces features. Therefore if org A can produce the same feature at half the cost of org B, then org A is twice as good as org B! And if spending 1 engineer hour on some task today will save you 100 engineer hours in the next few weeks, then you have just improved your org’s economics by 99 of those expensive engineer hours!

The fundamental and deadly flaw in this model is that it does not account economically for the speed of work through the engineering org—or what I’ll refer to from here on out as “latency”—the wall-clock hours, not paid engineer hours, that it takes the engineering org to turn some concept into reality. In other words, we can’t simply think of an engineering org as “an engine that produces thing X at cost Y.” We have to model it as “an engine that produces thing X at cost Y with latency Z,” and recognize that “latency Z” itself can and should be translated into some cost / value structure.

This is not to say that engineering leaders who employ this cost center model don’t care or think about latency. To the contrary, they often talk about it quite a bit, exhorting their teams to feel a “sense of urgency” and to exhibit a “just git ‘er done” attitude—but they treat latency as a moral or personal question—a matter of character or work ethic—rather than something that is, at its heart, organizational and economic.

It’s human nature to experience paid engineer hours as expensive and latency as annoying, because the costs of latency tend to be invisible—they usually take the form of lost opportunities or earnings, many of which, once you miss them, you never even know existed—rather than real, painful checks that you have to cut each month for payroll.

Consider an analogue: the rent your business pays on an office building. If you found a building that was only half the rent, you might well be tempted to move and count that as a huge savings—but that’s rarely the whole economic story. Is the new building farther away from where the bulk of your employees live? Does it lack the public transit options of the more expensive building? How’s the light? What’s the layout like? All of these factors can affect the amount of time your employees spend in the office, the amount and quality of work they get done there, and even the kind of people who want to work at your company in the first place—and if the cheaper building leads to a drop in productivity, or to worse hires, then that “savings” on rent might turn out to be very expensive indeed to your business’s bottom line, even though—and here’s the horrific part—that connection will probably never show up on your company’s balance sheet. It’s not hard to imagine the employee who found the cheaper building being rewarded with a fat bonus in the same cycle that a bunch of other employees are dinged for a stagnant product, increasing bug count, and flagging sales—even if all those problems were caused, to some extent, by the change in location.

One method to expose some of these invisible economic effects is to take them to an absurd extreme. For example, if your business is currently paying a half million in rent a year for a Boston office, with a workforce who lives in nearby suburbs, it’s clearly not a smart economic decision to move to a snow-cave in Juneau, Alaska—even if it’s wired for Ethernet and your annual rent would drop to $1. We’ve managed to magnify the invisible costs to a size where they can’t be easily ignored.

So let’s employ the same technique—reduction to some absurd extreme—in a thought experiment designed to demonstrate how the latency of your engineering org is almost certainly its dominant economic factor—much, much larger than the piddling six-figure salaries you’re paying the engineers it comprises.

The Thought Experiment

Role change: you’re no longer an engineering leader overseeing Facebook for Cats integration. Now you’re the CEO of a company that makes its money through big, enterprise contracts. A potential customer you’ve been after for a while is entertaining bids on a project, and will consider proposals—which are expected to include a working proof of concept—in one month.

You aren’t the only company trying to land this contract—there are lots of smart competitors. And, by the way, you’re not allowed to deliver early, even if you finish the proof of concept early—all proposals will be considered on the same day, one month from now.

As CEO you have two engineering teams available to you.

The first team is a group of good, steady developers, who correctly estimate that the proof of concept will take exactly one month for them to build (of course they can’t possibly know this, but that’s a story for another article and here we’ll just pretend they can, because we’re in a thought experiment and we can do whatever we want). Over this month of development, this team will cost the business $100,000 in salary and other compensation.

The second team, on the other hand, is a group of freelancers who are amazingly, inhumanly fast: they can produce the same proof of concept, at the same level of quality, in just one second. Before you get too excited thinking about all the money you’re going to save with this team, however, you should know this: for that one second of work, these freelancers will be invoicing you dearly—to the tune of $100,000.

Recapping your options, you have:

  • the normal team, which will take a month to produce the proof of concept for a total cost of $100,000

  • the insanely fast team, which will take a second to produce the proof of concept for a total cost of $100,000

The costs of the proof of concept are equivalent with either team, as is the quality of the product—only the latency differs. Obviously if you could deliver the proposal as soon as the proof of concept was done, you’d choose the insanely fast team every time. But that would be too easy, so in our thought experiment—where you’re not allowed to deliver the proposal early—does the latency even matter?

There’s only one scenario to consider with the normal team—they have to start working today, and they’ll finish just in time for the presentation. Start them even a day late, and they won’t finish.

With the insanely fast team, on the other hand, you have on the order of 2,592,000 scenarios to consider, as they could start and finish at any second in the entire month. But are any of these scenarios valuable?

Let’s take a look at a couple of these possibilities.

The Need for Speed

One obvious approach with the insanely fast team would be to produce the proof of concept immediately, in the very first second. Does that buy you anything? You can’t deliver the proof of concept early, but now that it exists, there are a couple things you could do with it.

For example, you could show it around and get a reaction—internally, if your business has some good proxies for your customer’s needs, or to one of the customers “on the ground” (not the Big Important People you’ll be pitching at the end of the month, just regular workers). Then you can take their feedback and do any of the following:

  • Iterate: Have the insanely fast team produce a second, improved version of the proof of concept—you’ll have to pay them another $100,000, but you’ll have good information about whether that’s worth it or not. You can repeat this process as many times as you like or can afford, and go into the demo having iterated through N versions to your competition’s one.

  • Abandon: If the feedback you get is “this is crap, and the only ways to make it good enough are too difficult or expensive to consider,” then you can abandon the contract and move on to try to sell something different to a different customer—or something different to the same customer! Meanwhile, your competition is sweating away trying to produce their own proofs of concept—squandering precious time and attention on a contest you already know isn’t worth winning.

  • Sell to Someone Else: By the rules you can’t deliver your proof of concept early to the one potential customer, but nothing says you can’t go out and try to sell it to a different one, or a different six. By the time proposal day arrives, you’re already a month ahead of your competition in other markets, and you might even have a nice story to tell about how your customer’s competition has already bought your version—and they’d better too, if they don’t want to fall behind.

So yeah, you could definitely say there’s some value to being able to finish the proof of concept in a second. That insanely fast team is starting to look pretty good right about now.

But wait…there’s more!

The Genius of Procrastination

What if you went to the other extreme, and waited as long as you could to produce the proof of concept, until the last possible second—literally while you’re walking down the hallway to make your presentation? Does that give you any interesting advantages?

One possibility that leaps to mind: given that your development is so expensive, you could do some cheaper exploration before you committed to a proof of concept. For example, you could send some PMs to shadow the customers, research companies that had tried similar approaches, etc.

By the time you commit to spending $100,000 on the proof of concept, you can have much better information about what it should do and what it shouldn’t. Maybe it turns out to be so difficult that you decide not to even build. Or maybe, with the insanely fast team at your back, an offhand remark as the customer is walking you to the presentation room prompts a quick phone call and a development cycle, allowing you to produce a last-second revision that totally changes the game.

In essence, by waiting until the last second to produce your proof of concept, you have the chance to be roughly 29 days, 23 hours, 59 minutes and 59 seconds better informed than your competition (the actual amount of time will depend on the particular month, whether it’s a leap year, etc., which is left as an exercise for the reader).

Mix and Match

But the real power of the insanely fast team comes when you mix and match all the techniques above.

Step 1: Do cheap research until you have an idea of what to build.

Step 2: Build it instantly and loop back to Step 1, until you decide another iteration isn’t worth $100,000 (either because the proof of concept is now good enough, or because you’ve decided to scrap the project).

Step 3: Profit!

Finish Early, Start Late

What the insanely fast team gives you, in other words, is the ability to finish early or start late. In an environment where uncertainty rules and information is value—like software development—that allows for tremendously valuable information gain, because what you finish early tends to generate information, and what you start late tends to benefit from newly available information1. The poor old regular engineering team, on the other hand, has to start early and finish late just in order to get the work done by the deadline. Their labor can neither generate extra information or benefit from it as it becomes available.

So Which Team Do You Want, Mr. CEO?

By now it should be clear: although the two teams cost the same, and produce the same quality output, you would be crazy not to choose the insanely fast team and their drastically reduced latency2. In fact, you’d be crazy not to pay a steep premium, well beyond the normal team’s salaries, to use the insanely fast team, or even to keep them inactive but on retainer.

This is so important it’s worth calling out: if you’re any kind of rational, you would pay a tremendous amount of extra money to use the insanely fast team, which means that a reduction in latency equals money. Real, actual money—and usually a lot of it. In our thought experiment, for example, a smart CEO would gladly pay $1,000,000 to use the insanely fast team instead of the regular team if it meant a massively increased chance at a $15,000,000 project. A smart CEO would see that not as “spending” money—but as investing it—putting money out into the world in the reasonable expectation of having that money return, now increased by some multiple.

Once you start thinking of engineering dollars as investment rather than cost, the fallacies of the “cost center” model become glaringly obvious. The equation behind your org isn’t “engineer hours paid for features or saved engineering hours”—it’s “money invested in the expectation of more money.” Often the money invested is in the form of paid engineer hours, but sometimes it’s new machines, or better chairs, or office space for a remote contingent, and so on. And sometimes the “more money” you expect in return comes from features for which customers will pay, but often (as in our thought experiment) it comes in the form of valuable information, or—if you’re doing it right—a reduction in (or prevention of) latency for future work, which, as we’ve just shown with our thought experiment, is worth actual money.

Sitting Down Again with Cindy and Scott

Let’s rewind back to that coffee with Cindy and Scott, where you as engineering leader were explaining to them all about how engineer hours could only be spent on features or efforts that would cut future engineer hours. With the clearer economic picture in mind, this argument no longer seems so simple and rational.

Cindy wanted time to work on DB deploy scripts, since she was the only one who could reliably get changes out to the production DB and was spending a chunk of her mornings doing so. At the time, what we heard behind her lament was “I’m getting bored doing the job you’re paying me to do and I need to be gently cat-herded to keep doing it”—but what we should have heard was “DANGER, WILL ROBINSON—a queue is forming in your engineering org.”

Cindy has become a bottleneck for changes making their way to production, and a queue of people trying to make those changes is forming behind her. Queues are one of the clearest signals of developing latency. What happens if Cindy is out for a few days on (gasp) vacation? No changes will go out. What happens if she becomes overloaded with other matters, and—without telling you—starts applying DB migrations only once a week, to “batch things up” and “be more efficient” with her time? Your latency has just skyrocketed invisibly—and the fact that this is possible should terrify you as an engineering leader. Cindy’s complaint is a warning of latency to come, and you need to nip that in the bud with extreme prejudice. You should probably allow Cindy to do her migration project—and you should definitely explain to her why you’re allowing it.

As for Scott, who wanted to rewrite the Frobulator Service from horrific PHP to stunning Scala because product had promised the time to clean it up: the “promise” from product is clearly economically irrelevant, and big rewrites tend to be a terrible investment, so you probably shouldn’t say yes to Scott’s exact request—but you still have some digging to do here to figure out whether this (almost certainly misguided) desire to rewrite is just a blue-sky engineering itch, or a signal that the Frobulator Service is creating latency.

First of all, Scott was only in that code to “update copyright years”—he wasn’t making functional changes, and apparently hadn’t made any in at least a year. Is this a clue that the Frobulator Service doesn’t see that much coding activity? Worth digging into, because if engineers aren’t touching the Frobulator Service because it’s frobulating3 just fine and there aren’t really any changes to make, that’s great—the code might read like Cthulhu’s diary, but it’s not affecting your latency and can be left as is for the moment. If, on the other hand, there are tons of changes that should go into the Frobulator Service, but which are finding their way into compensatory hacks throughout the rest of the codebase instead—because engineers are terrified to touch the Frobulator Service code—then you’ve got a brewing latency problem that you need to expose and deal with, because those hacks are probably already slowing you down, and the situation is only going to get worse. Almost certainly you still don’t want to commission a full-on rewrite, but a steady, incremental investment in testing, monitoring, and refactoring the Frobulator Service might be indicated.

Takeaways from Cindy and Scott

One of the deadliest things about latency is that often the slowdown of even a single piece of your org can introduce it, while making things faster generally requires steady work on a lot of fronts. That’s an imbalance that’s not in your favor. Add to this the certainty that latency is developing in your organization at every moment—that is the nature of organizations—and that it is often invisible to you (or any single individual)—and that, as we saw in our thought experiment, latency is tremendously expensive—and the response that’s indicated from you, the engineering leader, is a calm but constant terror.

Your job is to translate that terror into a form of shared vigilance: listen carefully to your engineers, dig into the problems they bring you, and ensure that every one of them understands the cost of latency and is on the lookout for it, making micro speed-ups everywhere they see the opportunity and surfacing brewing slowdowns.

In other words, make latency something your whole team seeks, hates, and destroys.

How to Invest in Latency Reduction

All right,” you say, “I’m convinced—latency is a bigger deal than I thought before, and something I can improve—in theory. But how do I do it in practice? I’ve made all those investments that didn’t help at all—how do I know that if I invest in something, it will actually improve my latency?”

Some of this also comes down to how much you invest, but we’ll leave that until Part II, and here just discuss what you can look to invest in.

Here are a few places you can start.

Activities Engineers Bitch About

Engineers tend to experience latency centers as painful or “busywork.” For example, do your engineers play “Rock Paper Scissors” to determine who has to spin up a new server? Does the loser go off cursing his luck and the world? Do your engineers go to absurd lengths to pack new services onto old machines, even when a new server would be the natural solution to the problem? Then take a look at what it requires to spin up a new server, and whether you can make an investment to make it less painful—you’ll likely effect a drop in latency.

Things Only Cindy Can Do

We saw an example of this with Cindy, who was the only engineer who knew enough about the prod DB to get migrations out. If only person X can do thing Y in your organization, you’ve created a bottleneck, and bottlenecks lead to latency. Cross-train or create tools to terminate these bottlenecks with extreme prejudice4.

Look for Queues

Queues are a manifestation of latency, and once you can see them, you can attack them. Find them where they’re visible—ticketing systems and so on—and try to make them visible where they’re not, using techniques like a Kanban board.

Automated Tests

Good automated tests reduce latency, because they help you make changes more quickly and confidently5.


Good monitors reduce latency, because they allow you to release more frequently, confident in the knowledge that, if something goes wrong, you’ll find out immediately6.

Post Mortems

A good post mortem is a great opportunity to let reality point you towards improvements that not only make your systems safer, but reduce your latency as well. Do them!

Decentralization with Safety Nets / Impact Reduction Schemes

Organizations often insist that high-impact changes to products or systems pass through multiple steps of centralized review for correctness, which can become a source of dramatic latency—sometimes on the order of weeks or months. Usually these controls exist for a reason, because the mistakes they attempt to prevent are expensive.

You can attack such a situation in two ways: either by making it harder to break things in the first place (often more difficult and expensive), or by changing the game so that breaking things isn’t as big a deal (often cheaper and easier). For example, if engineers can deploy potentially high-impact changes at will to a small percentage of traffic, or to a known beta-tolerant population, or to internal users, then the downside of breaking changes is capped, and is often eminently worth the decreased latency you enjoy.

And Many, Many More

We’ve only scratched the surface here: tools for operators, intelligent development tools, even crazy things like DSLs for demo or test data creation can all reduce your latency. Once you start looking specifically for projects that reduce latency, you will see opportunities everywhere.

How Not to Invest in Latency Reduction: REWRITE ALL THE THINGS

The “rewrite reflex” exhibited by Scott is, unfortunately, a real and dangerous tendency that almost all engineers have to some extent (I myself struggle with it daily): the fanatical belief that, if a system were rewritten to framework X or language Y, development would proceed much more quickly. Generally this doesn’t pan out, both because of the astounding (and routinely underestimated) cost of the rewrite, but also because the causes of latency introduced in real-world engineering are rarely addressed more directly by languages and frameworks than by operational and organizational changes7. The latency caused by having to write three ugly lines in one language rather than one pretty line in another tends to pale in comparison with delays in deploys, finding and fixing bugs that tests could have caught, etc. (note: I’m not arguing that there is no difference in language productivity, and no point to choosing a language for a new venture carefully, just that for a working system the gain is usually dwarfed by the rewrite cost and other, lower hanging fruit).

Incrementalism FTW

Maybe it’s a “one ring to rule them all” deployment system, or a templating system to speed up writing your views, or a monitoring framework to end all monitoring frameworks—whatever it is, if you think it will reduce latency, and it’s a big project, you should probably try breaking it into smaller increments, each of which reduces some latency, and release those independently, as each is ready.

Most engineers will hate to hear this. They’ve already “seen” the full system in their head, and now want to bang it out in a couple caffeine-fueled weeks. Typically if you object and request smaller increments, they will point out that, broken up into discrete releases, the job will require more hours overall, and therefore represent an inefficiency. They’re generally right, of course, that you will spend more engineer hours by delivering in increments—they’re just wrong about the economic consequences.

You should insist on smaller, incremental latency improvements, not just because of all the normal, eminently true reasons that big increments are bad (everything that makes waterfall a bad idea applies here too), but because latency reduction improves the same channels by which you deliver future latency reduction. That is, since latency reduction efforts generally come in the form of new software or processes, and what they’re reducing is the latency of delivering new software or processes, finished latency reduction efforts tend to speed up future latency reduction efforts.

Latency reduction is therefore a form of compound interest, which Einstein himself called “the most powerful force in the universe8.” Latency reduction works just like your retirement account—steady, incremental investments generate more value than infrequent, bigger investments, because you earn interest on your interest—so you want the money in the account as soon as it becomes available. When you break a big, massively valuable latency reducing project into numerous smaller (but still latency reducing) projects, some of which can be delivered earlier, the one-time multiple you pay on extra engineering hours is nearly always a rounding error compared to the benefit of compounded latency reduction you enjoy forever.

So Much for the Easy Part

All right, we’ve skirted the hard part long enough. At this point we understand some of the costs of latency. We’ve sounded out whether projects like those that Cindy and Scott want to undertake will actually reduce latency, talked about some other projects that are good candidates for reducing latency, and understand how to generate the maximum overall value by attacking them in valuable increments. But there’s still the small matter of that endless stream of features—how do we compare the relative value of a feature and a project to reduce latency for the delivery of future features, and prioritize appropriately? How do we know how much time to spend on latency reduction vs. features? And—more difficult still—how do we convince the CEO and other Important People in the business, who are the ones asking for those features and signing our checks, that they should allow us to carve out that time to work on latency reduction?

Tune in as we tackle that in the upcoming Part II: Selling the Big Boss.

  1. For more on this, see Reinertsen’s Principles of Product Development Flow—yup, it wouldn’t be a Hut 8 Labs Blog without a mention of that classic—but seriously, all joking aside, just go read it now. 

  2. Oh, and just in case you’re thinking “sure, if I could reduce my latency to a second, that would be one thing, but that’s crazy and extreme and impossible”: an engineering organization that managed to go from shipping simple improvements and bugfixes only with quarterly releases to being able to ship them in an hour (a realistic improvement that many organizations have already accomplished) would be seeing about a 2000X reduction in latency for those improvements and bugfixes, and that’s not even the upper bound—better testing, monitoring, and other investment can also drastically speed up what an engineer can reliably get done in that hour. 

  3. What? It’s a perfectly cromulent word. 

  4. Note: sometimes Cindy defines her value as “being the only person who can do X.” Helping her redefine her value more broadly is a key part of the leadership function, but a topic for a different article. 

  5. Bad tests can actually increase latency, because they over-specify implementation without adding any safety—but that’s a topic for a different article. 

  6. Bad monitors can actually increase latency, because they overwhelm and desensitize the people looking at them with irrelevant information or over-zealous alerting. But that’s—you guessed it—a topic for a different article. 

  7. And because frameworks are a form of moderate evil that is on occasion the lesser of two evils—but we’ll leave that for another time. 

  8. Or fine, maybe he didn’t. But look, whether Einstein said it or not, compound interest is pretty damn powerful, OK

Read the whole story
3734 days ago
Share this story
Next Page of Stories