tag:blogger.com,1999:blog-234303152024-03-20T08:07:33.752-07:00John Ratcliff's Code SuppositoryA place where I insert my code into the anus of the Internet.John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.comBlogger141125tag:blogger.com,1999:blog-23430315.post-57773823161531274012022-08-14T12:16:00.000-07:002022-08-14T12:16:18.024-07:00The Lightning Network In The Old West<p> <span style="font-family: Arial; font-size: 26pt; text-align: center; white-space: pre-wrap;">The Lightning Network ELIDHDICACS</span></p><span id="docs-internal-guid-201c6e65-7fff-3c5f-262a-068a7ceda0c2"><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"><span style="font-family: Arial; font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">(Explain Like I Don’t Have Degrees in Cryptography and Computer Science)</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"><span style="font-family: Arial; font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="border: none; display: inline-block; height: 485px; overflow: hidden; width: 624px;"><img height="485" src="https://lh3.googleusercontent.com/rBeAnJBy_yj1tmfCPLcnSDpX_Ovu9OonvcQzNf0m8iEl1r_iaQECQ_0hAQYtsauNvnnXLgw6lU3HHuGlnqFBiVuW9EoL17faN6Uuqxnb-X6hfHzWDs3-ZzjRSxpa7rTrChJ5Ht_ZSuH7XloupKRqNw" style="margin-left: 0px; margin-top: 0px;" width="624" /></span></span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This article will attempt to explain how the Lightning Network operates, by making extensive use of analogy and avoiding intricate details of the underlying cryptography which validates the entire system. The goal here is to give the layperson a general understanding of how the Lightning Network operates by using, as close as possible, a real world analogy. Needless to say, such an explanation is going to leave out a lot of important details; which are already well covered in </span><a href="https://lightning.network/" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">the Lightning Network technical documentation</span></a><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. At times, the metaphors used may seem a bit stretched since, in most cases, the real world analog would be wildly impractical. By using analogies we are attempting to describe operations which happen on computer networks capable of executing contracts hundreds of thousands of times a second and with perfection.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">So, let us begin by imagining a small town where everyone carries around gold in their pockets to buy and sell everything. It should be clear that such a situation would work just fine, since it has worked in human society for thousands of years. There is the question, however, of just how difficult it is to carry this gold with you everywhere you go; because of security concerns, and the challenge of constantly having to make small change all of the time. For this example, let us imagine that we do what early banks did during the days of the Wild West.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Instead of carrying around gold with you everywhere, you could deposit your gold in the bank and carry sheets of paper which represent it; paper which is redeemable for actual gold at the bank as well. As we all know, this is how things used to work back in the Old West. In fact, individual banks would each issue their own gold certificates of deposit, which could then be exchanged for goods and services around town. Since everyone in town would accept these notes as ‘good as gold’ this method functioned well for many years without people having to resort to carrying heavy gold with them everywhere they went, and constantly trying to make change by carving it up into ever smaller amounts. It is important to note, that these paper certificates could be redeemed for actual physical gold at the bank; payable to whomever presented it. Since anyone could redeem these notes, it didn’t solve the basic security issue of simply being robbed. In contrast, the Lightning Network solves this problem of security very well.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Now, using this analogy, let’s try to understand how the Lightning Network operates. To be clear, at first glance it may seem dramatically more complicated than the Old West system, but keep in mind that things which are complex to you and I are not necessarily an issue for computers. A modern computer can multiply, add, subtract, or divide any two arbitrarily complex numbers about a billion times a second or so; something no human could ever do. </span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In this analogy the following holds true:</span></p><br /><ul style="margin-bottom: 0; margin-top: 0; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Bitcoin represents the Gold</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Bitcoin network and blockchain represents the Bank, but, unlike a regular bank, it cannot practice fractional reserve banking (lending out more money than it holds). Every single claim represents actual gold (i.e. bitcoin) in 100% perfect and immutable reserve. So, in our fictional Old West town, we have something which also doesn’t exist in the real world, a perfectly honest and incorruptible bank with 100% matching gold reserves.</span></p></li></ul><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The following properties are true in this analogy: The bank is the absolute arbiter of truth; the bank holds all of the gold; the bank decides who gets the gold and in what distribution; the bank has absolute power when determining between truth or lies; you cannot lie to the bank because this bank knows with absolute certainty, whether a signed contract is true or untrue—something that cannot be forged. This is a lot to accept at face value, but these points are all actually true on the bitcoin network due to the power of cryptography and can be </span><a href="https://en.wikipedia.org/wiki/Digital_signature" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">mathematically proven</span></a><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. The details of this proof include a lot of nasty technical points that we are trying to avoid bringing into the presentation. So, let’s forego all the mathematical details and keep things simple. If you want to understand more thoroughly what makes these things possible, feel free to take an advanced course in cryptography at your </span><a href="https://crypto.stanford.edu/" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">local University</span></a><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In our fictional Old West town, things work a little bit differently with the Lightning Network model as a payment system. There is no paper money which is freely exchanged among businesses. Also, the ‘certificates of deposit’ that anyone can redeem at the bank don’t exist in the Lightning Network model as well. Instead, there are legal contracts between individuals; legal contracts which are enforced by the bank using a very strict and ironclad (programmatic) set of rules.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="border: none; display: inline-block; height: 517px; overflow: hidden; width: 624px;"><img height="517" src="https://lh5.googleusercontent.com/YaKKZouYBHZCaLzj9GApulG8NBs1KvtMne1c_UefdVkgYHsMbokEAs9TkE5F5KK6XmXpCY2v8NTAkgR54Qn80s_QoNljMapckqmRUVtjRga_EFMAbuYPOaVR31UZ1Eyl7gG6ScBGifT-IoVsa4KDrg" style="margin-left: 0px; margin-top: 0px;" width="624" /></span></span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Let’s say that your name is Bob the Farmer, and you want to do regular business with the town's General Store. Under the Lightning Network model, Bob the Farmer and the General Store would enter into a contract with the bank. First, they would each deposit a certain amount of gold at the bank. Let’s say Farmer Bob deposits $100 worth of gold to do business with the General Store. In this first example, the General Store doesn’t deposit anything, but they could if they wanted to. Both Bob and the General Store agree to a contract, with a particular set of rules, that will be enforced by the bank. Before they leave the bank, the two parties exchange an initial receipt with each other, a receipt stating the initial balance, with $100 belonging to Farmer Bob and, at this time, nothing to the General Store; we’ll talk more on this later.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Once they have opened up this joint account at the bank, Bob can now start buying goods at the General Store; up to the $100 worth of gold that he deposited at the bank to begin with.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="border: none; display: inline-block; height: 604px; overflow: hidden; width: 624px;"><img height="604" src="https://lh3.googleusercontent.com/v0anO4dHAQVGrUqfuj2sV8srJMaF1hYUjSd92PmQ-5sP_DFZLd1XS9McRxPYMNI3JJJDFRnLIfbcbEpRc1WBbsnV3F17RSuDGP2RUsmwiN4z5I7_SZbK7mVYP9oz95a1Qxx4k0xHDg_AO5nMWM-31g" style="margin-left: 0px; margin-top: 0px;" width="624" /></span></span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">On the first day, Bob goes to the General Store and he buys a hammer worth $5. Rather than Bob handing over $5 worth of gold, or even paper money, he signs a note that says, “Of the $100 I have deposited in the bank, $5 now belongs to the General Store, and $95 belongs to me.” Both he and the General Store sign the other’s copy of this receipt, and they each keep a copy for themselves. Farmer Bob’s copy of the receipt is signed by the General Store and the General Store’s copy of the receipt is signed by Farmer Bob. Neither of them sign their own name yet, they wait to do that until they present it to the bank to close out the account. As you can see, no actual money has changed hands. Instead, they have simply made a notation in their personal ledgers. Also note, that the bank </span><span style="font-family: Arial; font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">knows nothing about this transaction</span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, nor does the bank need to know anything about this transaction either. In fact, even if Farmer Bob and the General Store perform a thousand transactions, all the bank cares about, or needs to know, is the current final balance between them.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The next day, Bob goes back to the store and buys some nails for $1. Now, the same procedure happens as the day before. Two copies of a </span><span style="font-family: Arial; font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">new receipt</span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> are written up, that states that the General Store gets $6 of the $100 deposited at the bank, and Bob gets the remaining $94. Each party signs and dates the other’s receipt, but they don’t sign their own copy yet.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Now, here is an interesting point. The note from the day before, which stated that the store had $5 and Bob had $95, </span><span style="font-family: Arial; font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">can be thrown away!</span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> It is no longer needed. Other than for personal accounting purposes, this signed note with the old balance can be thrown in the trash since it no longer reflects the currently agreed-upon balance, and is null and void (exactly how it becomes voided will be discussed in more detail later). </span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">At any time, either Bob or the General Store can close their account with one another. They each have a valid signed copy of the balance, and the bank will honor it because, remember, the bank has the awesome super cryptographic power of knowing with absolute certainty that the signatures are correct. For reasons of security, the party that wishes to close the account does not sign their copy of the receipt until they are at the bank. The bank verifies that the signatures are valid, and divvies up the gold they were holding on behalf of the clients, according to the signed agreement on the note (this doesn’t happen immediately, we will discuss this later). The bank doesn’t need to know the transaction history between Bob or the store. The bank really doesn’t care. All the bank needs to know, is that the final balance signed by both parties, is valid.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The bank isn’t a charity. For the service of opening and then closing the account, they will charge a small fee. However, for the purposes of this discussion, fees are not particularly relevant; it just means that when the bank settles up the account, they might give the General Store $5.95 (instead of the $6) and Farmer Bob $93.95 (instead of $94), keeping the 10 cent difference as a fee for providing the service of executing the contract and holding the gold. Of course, the General Store could choose to pay the entire ten cent fee as the cost of doing business if they want. The two parties can simply agree to the appropriate fee when they set up the account..</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Hopefully, with this analogy, you can understand how this system could work. So long as the General Store and Farmer Bob maintain a current balance between the two of them, they: (a) don’t need to actually exchange any money (just signed and dated receipts representing the current balance); and (b), the bank doesn’t need to know about any of the individual transactions between the two parties. The bank only needs to know the final balance to settle up the bill at the end. It does assume that the bank is 100% trustworthy and infallible—which, as we know in real life, is an absurd assumption. However—and this is the amazing part—with the power of the bitcoin blockchain and smart contracts, this actually is completely true!</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Each time Bob buys something at the General Store, and exchanges a copy of the receipt, each party signs the other one’s receipt. However, it is very important to note that they do not sign their own copy yet! The reason for this is because once a receipt has been signed by both parties, it can be presented to the bank, by anyone, and it will be executed. If Bob signs his copy of the receipt right away, then what happens if he loses it? Let’s say Bob signs his receipt and drops it on the ground. If someone else finds it, that means they can present the receipt signed by both parties to the bank to initiate a settlement of the funds prematurely. It is important to note that this 3rd party cannot, in any way, steal any of the money. The contract with the bank states that Bob gets his share of the money and the General Store, theirs. By presenting the receipt, the bank will begin to close the account but will not, and cannot, redirect the funds to someone else. So, why would a 3rd party present a signed receipt if they can’t benefit financially from it? The answer is simple — even in the Wild West, some people are just jerks. </span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This is the reason neither party signs their copy of the receipt until they are actually ready to present it to the bank. A receipt signed by only one party is completely safe from meddlesome people who might manage to steal a copy. In fact, not only is it safe, but Bob could even make multiple copies of each receipt and store them in different locations. The receipts are extremely valuable to Bob and the General Store, because if either of them loses their receipt, they can no longer initiate a settlement transaction with the bank. So, for the purposes of backup and redundancy, both Bob and the General Store can freely keep multiple copies of their receipts just to be completely safe and secure; they are protected even if they hand those receipts over to a 3rd party for safekeeping.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">By now, this thought may have occurred to you: What is to prevent Farmer Bob from presenting one of the older signed receipts to the bank? In fact, what if, after receiving $100 worth of goods from the General Store, he tries to present the initial receipt that was exchanged back when the account was originally opened? How does the bank know that they should not honor this one? Here is where things get a bit tricky. If either party requests to close out the account, the bank will not pay out the funds immediately, even though the receipt is properly signed. Before they honor the receipt, giving Bob money he doesn’t deserve, the bank will first contact the General Store and ask, “Hey, Farmer Bob is trying to close out his account with the following receipt. Is this cool?” The bank continues, “Farmer Bob says that the last time you two did a transaction was on this date and he doesn’t owe you anything.” At this point, the General Store says, “Hell no, that’s not right! The most recent balance between us, is that he owes me $100 and (through the magic of cryptography) I can prove it!” The General Store presents the most recent signed receipt and the bank says, “Yep, you are right. Bob is a stinking liar and scam artist. For attempting to defraud you, he forfeits the entire $100; you get it all!” </span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 16pt; margin-top: 0pt;"><span style="color: #666666; font-family: Arial; font-size: 15pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The penalty for submitting an invalid receipt, is the complete loss of all funds, which are handed over to the counterparty.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This is probably the most critical part of how the system works. Even though anyone can present a valid receipt to the bank at any time, the bank will wait a short period so that the counterparty can contest the receipt, simply by presenting proof of a newer one. Remember that Farmer Bob cryptographically signed the newer receipt, and his signature confirms that this transaction is newer than the bogus one he presented to the bank. Again, this is an example of something that would be almost impossible in the material world, but can be irrevocably proven using cryptographic systems. Without this window of time for the counterparty to object, this entire system would not work. The bitcoin network is the world's most powerful time-stamping system and the Lightning Network leverages against it to make this kind of a payment system an actuality.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Considering the penalty for presenting a false receipt to the bank, Farmer Bob is highly motivated not to lie to the bank.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">At this point, we should probably note that this analogy wouldn’t work very well in the real world because, in the real world, being able to perfectly validate dates and signatures, with 100% certainty, is not so easy. Yet, with computer networks and the power of cryptographic signatures, this is entirely possible.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Now, let’s look at a couple more edge cases. What happens if Farmer Bob presents the invalid receipt, the General Store is notified, but they don’t say anything about it? In that case, the transaction will still go through since it has valid signatures of both parties—because no one objected. The time window, where the second party is notified, is what prevents people from defrauding someone else; it is because of the threateningly high risk of losing all of their money on deposit. In the real world, the bank notifying the store might come in the form of a phone call. In bitcoin, that notification comes in the form of a transaction published to the bitcoin network; this transaction is not valid until a certain amount of time has passed, but everyone can see and monitor it, including, in this analogy, the General Store. </span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">You might be wondering what would happen if Farmer Bob presents an invalid receipt only to have the bank fail to contact the General Store for confirmation. Then what? Who knows, maybe the General Store is closed for the holidays and Farmer Bob thinks he can take advantage of this to get his invalid receipt accepted. This is, indeed, a problem. If the counterparty fails to properly monitor any attempts to close the account prematurely with an invalid receipt, and misses the deadline, they could definitely be defrauded. With the Lightning Network, there are multiple redundant systems running 24/7, which monitor the closing transactions submitted to the bitcoin network in order to make sure that this cannot happen. The mechanism which enforces this is as follows. Each time the two parties update their balance receipts, they both create a side contract; we will call this the ‘claiming transaction’. This is a signed contract which is only valid if the counterparty attempts to close the account fraudulently. That is the only condition where this contract holds true; in fair account closure, there is no need to readdress this claiming transaction. Within this side contract, the counterparty agrees to pay a small bounty to whoever presents the contract to the bank. By sharing this contract with a group of bounty hunters, there is an incentive for them to police any attempts to commit a fraudulent closing transaction on your account, because they would get rewarded for doing so. If this were ever to occur, as it is a rare or infrequent event, then you can be assured that the claiming transaction ensures the securing of your funds; even if you were unavailable to do so yourself at the time.</span></p><br /><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">So far, we have presented an explanation for how two people could perform financial transactions without actually having to exchange any money directly; but what if Farmer Bob wants to pay someone who he does not already have an existing account with? If it is someone he regularly does business with, he could consider making a new deposit with the bank and transact with them the same way that he has being doing with the General Store. However, if Farmer Bob wanted pay Seamstress Alice just this one time, it is a lot of trouble to create a whole new deposit and contractual relationship with the bank for just one payment. Luckily, he doesn’t have to go to all this trouble. In this particular Old West town, for exemplar purposes, almost everyone does regular business with the General Store. Both Bob and Alice already have existing business relationships with the store. Therefore, Bob can pay Alice through the General Store as an intermediary. The solution is actually relatively straightforward. Farmer Bob can send $10 to Seamstress Alice by asking the General Store to forward it to her. First, Farmer Bob adjusts his balance with the General Store and, in turn, the General Store can adjust its balance with Alice for the same amount. As you can see, no actual money changes hands, but parties along the chain merely update balances between them to get the same effect.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="border: none; display: inline-block; height: 340px; overflow: hidden; width: 624px;"><img height="340" src="https://lh5.googleusercontent.com/TtJ2Aq46HJm-KEfg0Fog1InK67VsslKYgnFaLZ3LM8r8sbnIEuV_bkdTR_KWIJ011LyuIGTwNodEMHiSCpFSKSBZBGayAsHAJLuqHx_OLWmAM2pcIfHji6cxeDExld9lW6Xjg6Cc9KLhbAwXInQVUg" style="margin-left: 0px; margin-top: 0px;" width="624" /></span></span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">We can even take this scenario a step further. Let’s say that Alice DOESN’T have an open account with the General Store. However, let’s say that she does have one with the Blacksmith, who, in turn, has an account open with the General Store. In this scenario, Bob could pay Alice by adjusting his balance with the General Store, who then in turn, adjusts their balance with the Blacksmith, who then adjusts his balance with Alice. There is an often-quoted ‘</span><a href="https://en.wikipedia.org/wiki/Six_degrees_of_separation" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">six degrees of separation</span></a><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">’ theory, which posits that everyone knows every other person on the planet within just six logical connections. Likewise, with the Lightning Network, every person can pay anyone else within just one, two, or maybe three hops along the way. In fact, it is this process of connecting people along existing open payment channels, that creates the underlying ‘network’ that the Lightning Network operates upon. Using the Lightning Network, anyone can pay another simply by forwarding funds and without the need to create any new deposits at the bank. By adjusting everyone’s balance sheet in accordance to everyone involved, you can effectively move money around without moving any physical money at all.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">It may seem like this system is too highly complex and inefficient to equip everyone with the ability to pay everyone else; yes, in the real world, this would be the case. The hassle of having to update everyone’s outstanding balance just to ‘move some money around’ may seem like an impractical burden. However, just as multiplying a billion numbers in one second is fairly unrealistic for the average person to do, such a task is trivial for a computer. </span></p><br /><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">So far, we have avoided diving too deeply into the technical details about how this system actually works. We have been able to explain, through a fairly straightforward analogy, how two people or businesses can perform financial transactions securely without needing to exchange any physical money—so long as they have a deposit backing it up. Unfortunately, trying to explain how multi-hop payments (payments forwarded through intermediaries) can be done securely, through an Old West analogy, is not quite so easy to do. Therefore, the following explanation will act only as a very rough approximation of how the multi-hop payment system actually works.</span></p><br /><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Let us return to the simple example of Bob wanting to pay Alice, with the General Store as an intermediary. What is needed, is a way to guarantee Bob and Alice that his funds were properly forwarded and credited to Alice. The way this is accomplished is by creating a temporary side contract between each party. First, Alice </span><span style="font-family: Arial; font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">directly </span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">informs Bob of a secret; for the sake of this example, let’s say that it is a riddle question. Next, Bob creates a contract with the General Store and that contract includes the riddle as a dependent clause; the General Store can only keep the money he is holding onto for Alice if he can provide the answer to the riddle, but the General Store is unaware of the answer—Alice is the only person who can provide an answer to this riddle. When Alice attains the copy of the contract from the General store, she can redeem her riddle answer for the money. If there are multiple people in the chain to Alice, none of them will be able to obtain the money since none of them have the answer. Each person in the chain forwards the contract, with the riddle question as a dependent clause, until it reaches Alice in this example. Here is the sequence of events explaining this example:</span></p><br /><ol style="margin-bottom: 0; margin-top: 0; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Bob wants to pay Alice $10</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Alice gives Bob a riddle directly.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Bob discloses the riddle </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">question </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">to the General Store and asks the General Store to forward $10 to Alice.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Now the General Store creates a side contract for $10 with Alice, providing the riddle question in the terms, which </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">proves </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">that he got the money from Bob because the only way he could know the riddle question is if he got it from Bob.</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Once Alice confirms that the payment was forwarded from Bob, </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">she then gives the answer</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> to the riddle to the General Store, which fulfils the contract terms. Now, everyone is settled up (to be clear, what actually happens with the Lighting Network, is that instead of a ‘riddle’ and an ‘answer’, there is a mathematical equation to be solved containing two components; one component serves as the ‘riddle question’ and the other serves as the confirmed mathematical result).</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: decimal; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Once everyone knows both the riddle and the answer, this proves that the money was properly forwarded along the chain. At this point, the side-contract can be discarded and the two parties simply adjust their normal balance.</span></p></li></ol><br /><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Lightning Network is a system for exchanging signed contracts, which entitles various entities access to previously-deposited funds, in a safe and secure way. The bitcoin network acts as the perfect Judge. The bitcoin network can validate any contract with absolute authority and veracity. These contracts essentially act as a mathematical equation to be solved, and the bitcoin network can rigorously and perfectly enforce the terms of any contract submitted to it.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This, in a nutshell, is the genius of the Lightning Network. It is an ambitious and audacious application of ‘smart contracts’; contracts which are not enforced by a real-world judge, but are instead enforced by the power of the bitcoin trust network and impossible-to-hack mathematics.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Avoiding the Old West analogy for a moment, here is a summary of what the Lightning Network actually ‘is’:</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">●</span><span style="font-family: Arial; font-size: 7pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Lightning Network is an extension of the bitcoin network which uses time-locked, hashed smart contracts to safely exchange value.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">●</span><span style="font-family: Arial; font-size: 7pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">All Lightning Network transactions </span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">ARE </span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">bitcoin transactions. Admittedly, they are unconfirmed transactions. However, these are transactions which are dependent upon an already-existing and confirmed funding transaction that resides on the blockchain.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">●</span><span style="font-family: Arial; font-size: 7pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">By using smart contracts, Lightning Network ‘transactions’ are not transactions in the traditional sense. Participants are merely signing a contract representing an agreed-upon balance in an existing account. This can be done safely and securely. Not only that, but it also avoids having to publish every intervening transaction to the blockchain. This enhances privacy, security, and has a multiplier effect on the main bitcoin blockchain. It is important to avoid thinking about Lightning Network transactions as something separate from bitcoin transactions—they are not. They actually </span><span style="font-family: Arial; font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">are</span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> bitcoin transactions, and the only thing the ‘network’ part of the term ‘Lightning Network’ does, is facilitate the mechanics of safely/securely signing and forwarding these bitcoin transactions among participants of the system.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">There are some things that the Lightning Network does extremely well, and there are other areas that still have some rough edges to smooth out.</span></p><br /><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 16pt; margin-top: 0pt; text-align: center;"><span style="color: #666666; font-family: Arial; font-size: 15pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Things which the Lightning Network does well</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">●</span><span style="font-family: Arial; font-size: 7pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Lighting Network is capable of supporting billions upon billions of transactions for free (or nearly free), since the only thing that is happening between parties, is the updating of a balance sheet between two individuals, and revision of signed contracts. It does, however, require that the main bitcoin network has a sufficient transaction capacity equal to the number of individuals and businesses who wish to open and close channels. Currently, the bitcoin network blocksize restriction limits the actual number of people capable of opening and closing channels, to only a few million. The bitcoin network itself will, naturally, need to grow larger in order to support the expected influx of active users over time; it will need to grow large enough to allow every user of the Lightning Network to regularly open and close payment channels.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">●</span><span style="font-family: Arial; font-size: 7pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Lighting Network enables the transfer of value in extremely tiny quantities. In the real world, moving around value equivalent to one one-hundredth of a penny would be both wildly impractical and cost prohibitive. But, over the Lightning Network, this is a trivial operation and is essentially free. You might wonder what good is such a miniscule exchange? The reality, is that in our ever more connected world, there are not only billions of humans interconnected via their mobile devices and the Internet, but soon, there will be billions upon billions of ‘things’ connected as well. The ability for billions of devices to exchange miniscule amounts of value, essentially for free, is going to transform our world in a multitude of ways. For example, what if you could pay a fraction of a penny each time you listen to a song, watch a video, or read an article on the Internet? In this way, you would be able to directly fund the artist behind that work and, in aggregate, all of those fractions of a penny can really add up (@see ‘</span><a href="https://www.youtube.com/watch?v=6NjLTnI9uGM" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Office Space</span></a><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">’). In exchange for making these tiny payments, you could opt-out of obnoxious advertising without necessitating enrollment in some sort of ‘subscription’ service. You could pay for Wi-Fi service, or any other streaming service, ‘by the nanosecond’ if you wanted to. Simply pay for what you use, and it can all happen seamlessly over the Internet. Now, imagine that you are a musician who just created a new song. You might allow anyone to listen to your song for the low, low price of just one one-hundredth of a penny per listen. Maybe, your song turns out to be a really great and popular song. People dig the hook, and it just so happens that a few hundred thousand people listen to the song a few times each, for a total of one million plays. In return, the artist would receive $10,000 for just that one song alone! With no middlemen and virtually no fees, it would all come into his payment channel one one-hundredth of a penny at a time. And, who wouldn’t pay one one-hundredth of a penny to listen to a good song? As you can see, extremely high-frequency and low-value microtransactions enable a whole new way to pay for nearly everything on the Internet.</span></p><br /><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In conclusion, the Lightning Network is an incredible invention. It shows the almost mind boggling power of smart contracts; contracts which have mathematical certainty and are executed with absolute precision and security by the bitcoin network. Over time, systems based on this technology will completely and utterly transform the entire financial system of the planet and enable the ‘internet of things’; a world containing billions of interconnected devices capable of directly exchanging value and performing tasks and duties for the services they provide.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Lightning Network is not a competitor of the bitcoin</span><span style="font-family: Arial; font-size: 8pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">network in any way but, rather, it is an extension of the network itself—one that enables new kinds of value transfer, backed up by the most powerful decentralized trust system in human history!</span></p><br /><br /><br /><br /><br /></span>John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com0tag:blogger.com,1999:blog-23430315.post-5727987062880443442020-07-21T14:44:00.000-07:002020-07-21T15:02:48.725-07:00<br />
<div style="background-color: white; text-indent: 0px;">
<div class="separator" style="clear: both; color: #14171a; font-size: 19px; letter-spacing: normal; text-align: center; text-transform: none; white-space: pre-wrap; word-spacing: 0px;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAd3thM1sB-HAo32C1jKFso_KsbD9iv6zW3jA-ZqWHBoNunpkI72pcIsJPROMPwpis12bAUodP8fN9bcEZ3ilPmwJT7t_dnpngiZ_K1hftzx1qx1FouXg5GjVGwpdaG57IvHW47g/s1600/victoria.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="720" data-original-width="480" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAd3thM1sB-HAo32C1jKFso_KsbD9iv6zW3jA-ZqWHBoNunpkI72pcIsJPROMPwpis12bAUodP8fN9bcEZ3ilPmwJT7t_dnpngiZ_K1hftzx1qx1FouXg5GjVGwpdaG57IvHW47g/s640/victoria.jpg" width="425" /></a></div>
<div style="color: #14171a; direction: ltr; font-size: 19px; letter-spacing: normal; overflow: hidden; position: relative; text-align: left; text-transform: none; white-space: pre-wrap; word-spacing: 0px;">
<span style="direction: ltr; overflow: hidden; position: relative; white-space: pre-wrap;"><br /></span></div>
<div style="color: #14171a; direction: ltr; font-size: 19px; letter-spacing: normal; overflow: hidden; position: relative; text-align: left; text-transform: none; white-space: pre-wrap; word-spacing: 0px;">
<span style="direction: ltr; overflow: hidden; position: relative; white-space: pre-wrap;"><br /></span></div>
<div style="direction: ltr; overflow: hidden; position: relative; text-align: left;">
<div style="text-align: left;">
<span style="color: #14171a;"><span style="font-size: 19px; white-space: pre-wrap;">I am really proud of myself today, and of the entire development community as well. About 5 years ago I was given the assignment to port the PC game 'The Witness' by <a href="https://twitter.com/Jonathan_Blow">@Jonathan_Blow</a> to the NVIDIA Shield as a work project.
The game had several thousand lines of SSE code that I needed to port to the ARM/NEON instruction set.
Needless to say, this was not fun at all. I wondered if maybe I could do an automatic conversion and that seemed like it might be possible. So, I created a header file which would compile as SSE on Windows but as the NEON equivalent on ARM and then I wrote a bunch of unit-tests which verified that I got the same results for all intrinsics either way.
I only ported as many SSE intrinsics that the game I was working on used. So I did not have anywhere near full coverage of all of the SSE instructions. But it was a start and better than nothing. And I couldn't find any place on the internet where someone had done the same thing.
I did find a library which could convert NEON to SSE from an engineer at Intel named Victoria Zhislina (her picture in the title of this post).
Her library is located here: <a href="https://github.com/intel/ARM_NEON_2_x86_SSE">https://github.com/intel/ARM_NEON_2_x86_SSE</a></span></span><br />
<span style="color: #14171a;"><span style="font-size: 19px; white-space: pre-wrap;">
This proved extremely useful as a starting point but going from NEON to SSE is definitely not the same thing as going from SSE to NEON.
Once I got the code I needed for 'The Witness' ported, I had no other immediate use for this library. However, I figured it might be useful to other people as a starting point. So, I uploaded it as an open source project to GitHub with the approval of my manager. For the first year or so I know a couple of people were using it, because they added support for additional intrinsics and I merged them to the depot.
Then, I kind of forgot about it.
This morning I got an email from a dev who informed that someone had made a fork of my project and has been adding more and more features to it over the past few years. I did not know until this morning that had even happened.
Today that library has now been integrated into a number of projects and has had contributors from myself at NVIDIA as well as engineers from Amazon, Google, and Apple.
This is so very gratifying that I wrote a partial library, I didn't finish it, but others in the FOSS community made it so much better and shared it with everyone under the same open MIT license.
Unlike other snippets of code I have written in the past, which usually become obsolete pretty quickly, this library is liable to live on and still be useful long after my death. I'm quite pleased with this result.
Here is a link to the new official SSE2NEON project.
<a href="https://github.com/DLTcollab/sse2neon">https://github.com/DLTcollab/sse2neon</a></span></span></div>
</div>
</div>
<div style="-webkit-text-stroke-width: 0px; background-color: white; color: #14171a; font-family: system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Ubuntu, "Helvetica Neue", sans-serif; font-size: 19px; font-variant-caps: normal; font-variant-ligatures: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: pre-wrap; widows: 2; word-spacing: 0px;">
<div style="direction: ltr; overflow: hidden; position: relative; text-align: left; white-space: pre-wrap;">
<br /></div>
<div style="direction: ltr; overflow: hidden; position: relative; text-align: left; white-space: pre-wrap;">
<br /></div>
</div>
<div>
<br /></div>
<br />John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com0tag:blogger.com,1999:blog-23430315.post-47726860616631412452020-04-30T15:39:00.002-07:002020-05-01T08:46:55.712-07:00<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/Ui2wKoze91k/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/Ui2wKoze91k?feature=player_embedded" width="320"></iframe></div>
<br />
<br />
Just a short post today to cover some recent changes to V-HACD.<br />
<br />
The YouTube video above demonstrates how to use the DebugView and ConvexDecomposition application to experiment with V-HACD. This tool is Windows only.<br />
<br />
You can find the DebugView tool with the corresponding ConvexDecomposition application here:<br />
<br />
<a href="https://github.com/jratcliff63367/debugview">https://github.com/jratcliff63367/debugview</a><br />
<br />
If you go to the official github repository for V-HACD located here:<br />
<br />
<a href="https://github.com/kmammou/v-hacd">https://github.com/kmammou/v-hacd</a><br />
<br />
After you clone the depot go to this branch:<br />
<br />
"git checkout performance-improvements"<br />
<br />
This branch has not yet been merged to the master, as I don't want to risk that it might break builds for anyone else. I have tested that this compiles for both Linux and Windows though.<br />
<br />
With a high triangle count mesh and a very high voxel resolution (10 million) the performance branch clocks in at 7.5 times faster than the master branch.<br />
<br />
This branch has a bug fix and some performance optimizations made both by myself as well as Danny Couture from Epic. The bug fix was in the option called 'project hull vertices'. This option was added so that the output convex hulls would accurately conform to the original source mesh. Without this option enabled the output convex hulls lack sufficient precision due to the voxelization step.<br />
<br />
However, in the previous version, the project hull vertices option had a bug in it which <i>could </i>cause it to produce invalid results in some edge cases.<br />
<br />
The previous algorithm tried to 'project' onto the source mesh by performing a raycast and looking for the closest point. Most of the time this worked fine, but on some example meshes it would fail and produce bad output.<br />
<br />
The new version of the code builds an axis aligned bounding tree (AABB) from the source mesh and then uses a 'closest point' method to find the nearest point on the original source mesh to any given voxel position. I have yet to see this algorithm fail.<br />
<br />
Some other changes have been made to the source that is outlined here or you can read the comments in the header file:<br />
<br />
<a href="https://github.com/kmammou/v-hacd/blob/performance-improvements/src/VHACD_Lib/public/VHACD.h">https://github.com/kmammou/v-hacd/blob/performance-improvements/src/VHACD_Lib/public/VHACD.h</a><br />
<br />
<br />
<ul>
<li>The version number was bumped to 3.1 </li>
<li>An optional callback function was added to notify your application when the convex decomposition background task has completed when running V-HACD asynchronously. Applications can use this callback to raise a signal in their own application if needed.</li>
<li>Some general code cleanup was performed removing some dead or legacy code that, as best I could tell, was no longer being actively used. This included all of the OpenMP, OpenCL, and SSE code. None of which was being actively used and was just causing a maintenance burden in the code base. The other performance optimizations made in this build provide greater benefit than OpenMP/OpenCL and SSE were or could have provided. It also benefits for the source code to be vanilla C++. </li>
<li>Other code cleanup was removing the 'tetrahedral mode' which I don't think we actively being maintained or used. Also, some of the code used a template to instantiate it with either floats or doubles. Since V-HACD should always be run with double precision, this template was removed for the sake of simplifying the code.</li>
<li>A new optional interface was added called 'IUserProfiler' to give your application a chance to add profiling markers to measure the performance bottlenecks in V-HACD. This feature was added by Danny Couture at Epic as part of the UE4 integration of V-HACD.</li>
<li>Another new optional virtual callback interface was added called 'IUserTaskRunner' which allows the application to provide it's own threading hooks rather than using std::thread which is how it works by default. This option was also added by Danny Couture of Epic as part of the UE4 integration of V-HACD.</li>
<li>The previously mentioned bug fix to the project hull vertices option was added.</li>
<li>During the voxelization step some code optimizations were made, again by Danny Couture of Epic. These optimizations are detailed further in the header fiel.</li>
<li>Another optimization added to the voxelization process was this. Previously the code would voxelize the input mesh multiple times at increasing resolutions until it reached the desired target number of voxels specified by the user. This step was not needed. The voxel size needed is largely implied by the voxel resolution. So rather than voxelizing the input mesh multiple times, it is only done once based on the supplied voxel resolution value.</li>
<li>The previous version of V-HACD tried to support the option of rotating the input mesh when performing the voxelization step. However, this feature was not being actively used and had performance implications. So it was removed, and made the voxelization step noticeably faster.</li>
<li>Another optimization that was added was to try to run the ACD step in parallel. The ACD step recursively splits the source mesh along optimal splitting planes. Since this process can take a while the loop was refactored so that when considering a large number of source mesh fragments (known as a 'PrimitiveSet' in the code) it could evaluate the best splitting plane and apply it in parallel. A new piece of helper code was added called 'SimpleJobSystem' which is a very basic and non-sophisticated way to execute a large number of logical 'jobs' across multiple threads.</li>
<li>A new option called 'FillMode' was added which can control how the V-HACD algorithm determines which voxels are 'inside' the source mesh, which are 'on the surface', and which are on the 'outside'. If the source mesh is not water tight, the default FillMode (FLOOD_FILL) can fail. So a new fill mode was added called 'RAYCAST_FILL' which can correct for that issue. Another fill mode 'SURFACE_ONLY' was added if you want the convex decomposition to converge on just the 'skin' of the source mesh, making it hollow, as opposed to treating it as a solid object.</li>
</ul>
<br />
<br />
<br />John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com0tag:blogger.com,1999:blog-23430315.post-54465776219902224362018-11-02T12:22:00.002-07:002018-11-02T12:22:30.077-07:00STL tracker : Sample code which demonstrates how to intercept and track all memory allocations performed by STL containers<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiy0POgpP6zGvL6TVzEkPiWAmiKz512-40QMzKvzb6s3viUnwMvIEptSIHBacXpR47gOZ7duecfcC2_n5k77JekF2fW177X817opIAFjXxaFoA-MgzIDh9y4sPnjQOc1WJ7izRzQA/s1600/Bloodhound-On-White-031.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="487" data-original-width="729" height="427" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiy0POgpP6zGvL6TVzEkPiWAmiKz512-40QMzKvzb6s3viUnwMvIEptSIHBacXpR47gOZ7duecfcC2_n5k77JekF2fW177X817opIAFjXxaFoA-MgzIDh9y4sPnjQOc1WJ7izRzQA/s640/Bloodhound-On-White-031.jpg" width="640" /></a></div>
<br />
My friend Chris Dannemiller showed me the template magic necessary to make it so that you can intercept and track all memory allocations performed by STL containers.<br />
<br />
To use it, you simply include an alternate header file and instead of using something like: std::vector you use stdt::vector.<br />
<br />
This code has been tested to work with Visual Studio 2017, GCC, and CLANG.<br />
<br />
<a href="https://github.com/jratcliff63367/stltracking">Here is the link to the github depot with the demo code.</a>John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com0tag:blogger.com,1999:blog-23430315.post-63188849495169209202018-05-04T08:07:00.000-07:002018-05-04T08:07:32.259-07:00Hierarchy Builder : A code snippet to derive hierarchies from a randomized collection of bodies and joints<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKZurzPvADOOo_bvVIdQOgy2-4p5Vosd63xyDLCbMtfS4SrRV5eM5PuNYVzsZ4vP_WPhKZa9huR-p2rxFeWbZ_hy3g4dzsWHrzzDNqbi0Xj7pNXdAUW8eBf2hc9wFMhPoFQhOD8A/s1600/hierarchy.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="300" data-original-width="400" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKZurzPvADOOo_bvVIdQOgy2-4p5Vosd63xyDLCbMtfS4SrRV5eM5PuNYVzsZ4vP_WPhKZa9huR-p2rxFeWbZ_hy3g4dzsWHrzzDNqbi0Xj7pNXdAUW8eBf2hc9wFMhPoFQhOD8A/s320/hierarchy.jpg" width="320" /></a></div>
<br />
Today I am releasing a code snippet that I had to write for a work project. Since it was generally useful for me, I'm open sourcing it under the theory it might be generally useful for others.<br />
<br />
Here is the problem it solves.<br />
<br />
Let's say you need some physics data in an ordered hierarchy of child-parent relationships. However, the input data is unordered. It could even be completely randomized.<br />
<br />
Often times when you have input data which comes in as a collection of rigid bodies and joints which connect those rigid bodies, you do not have any guarantees about the order in which they were defined.<br />
<br />
If you need order, then you have to build a graph from the input data to derive that order.<br />
<br />
That's what this code snippet does.<br />
<br />
It can be found here on GitHub: <a href="https://github.com/jratcliff63367/hierarchybuilder">https://github.com/jratcliff63367/hierarchybuilder</a><br />
<br />
An example of how to use the class can be found here:<br />
<br />
<a href="https://github.com/jratcliff63367/hierarchybuilder/blob/master/main.cpp">https://github.com/jratcliff63367/hierarchybuilder/blob/master/main.cpp</a><br />
<br />
To use the class you first add all of your rigid bodies by unique name. Since this is merely a graphing exercise it doesn't need to actually know any of the rigid body properties, just that there is a rigid body of a particular unique id/name.<br />
<br />
Next, you add all of the joints which connect some, or all, of these rigid bodies. These joints could describe one, or more hierarchies. Each joint is given a unique identifying name (since it is valid to have more than one joint connect the same two rigid bodies).<br />
<br />
The rigid bodies and joints can be added in a completely randomized order.<br />
<br />
Once you have specified the joints and rigid bodies, you simply call the method 'build' to compute the hierarchies.<br />
<br />
The algorithm works as follows:<br />
<br />
<br />
<ul>
<li>First, it examines all of the joints provided and figures out which rigid bodies are not connected by any joints at all. Those are then identified as 'disconnected rigid bodies' which are not part of any hierarchy and can be treated as discrete entities.</li>
<li>Next, for each joint not currently represented, you see if it can be added to an existing hierarchy or not. If it cannot, then you add it to a new hierarchy.</li>
<li>You could end up with hierarchy fragments due to the fact that joints could have been added in completely random order. To correct for this you iterate through every pair of hierarchy fragments and see if they can be merged together. This process is repeated until no two hierarchy fragments can be merged into one.</li>
<li>Once this merge pass is complete, it then examines the output hierarchies and looks to see if any of the joints are 'loop' joints; meaning they connect back to an existing rigid body in the chain. For some physics engines loop joints may need to be special cased, so there may be value in identifying and flagging them.</li>
</ul>
<div>
<br /></div>
<div>
That's it. This is hardly the most complex algorithm or piece of code in the world. And the topic comes up infrequently. However, if you ever find yourself with an unordered collection of rigid bodies and constraints and need to derive their order, this code snippet could be useful.</div>
<div>
<br /></div>
<div>
The header file for the HierarchyBuilder class is here: <a href="https://github.com/jratcliff63367/hierarchybuilder/blob/master/HierarchyBuilder.h">Header File.</a></div>
<div>
The implementation file is located here: <a href="https://github.com/jratcliff63367/hierarchybuilder/blob/master/HierarchyBuilder.cpp">Implementation.</a></div>
<div>
<br /></div>
<div>
Please let me know if you find this code snippet useful.</div>
<br />
<br />
<br />John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com0tag:blogger.com,1999:blog-23430315.post-2239983990572813542017-01-20T14:19:00.000-08:002017-01-20T14:30:21.964-08:00SSE2NEON Bug Fixes and new Intrinsics supported<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbG9106CK1xObR4fpqA1mxfYsQ-WzD-zu4wXMH8RSINrVnvwwqu_EEVM26Cl8GeVy1Kesey9ZXcT1OtHYGHEX9B3D0wts3IqScfXAoJEWiBMTF1AOXHRBXQXMY2AN4oj0CwFybxQ/s1600/IMG_20160422_111159.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbG9106CK1xObR4fpqA1mxfYsQ-WzD-zu4wXMH8RSINrVnvwwqu_EEVM26Cl8GeVy1Kesey9ZXcT1OtHYGHEX9B3D0wts3IqScfXAoJEWiBMTF1AOXHRBXQXMY2AN4oj0CwFybxQ/s640/IMG_20160422_111159.jpg" width="480" /></a></div>
<br />
<br />
This post is just a quick announcement about a new version of <a href="https://github.com/jratcliff63367/sse2neon">SSE2NEON</a> with the following changes:<br />
<br />
The official github repo for SSE2NEON can be found here: <a href="https://github.com/jratcliff63367/sse2neon">https://github.com/jratcliff63367/sse2neon</a><br />
<br />
The unit tests have been refactored. They no longer assert on an error, instead they return a pass/fail condition<br />
<br />
The unit-tests now test 10,000 random float and int values against each intrinsic.<br />
<br />
SSE2NEON now supports 95 SSE intrinsics. 39 of them have formal unit tests which have been implemented and fully tested on NEON/ARM. The remaining 56 still need unit tests implemented.<br />
<br />
A struct is now defined in this header file called 'SIMDVec' which can be used by applications which attempt to access the contents of an _m128 struct directly. <br />
<br />
It is important to note that accessing the __m128 struct directly is considered bad coding practice by Microsoft: @see: <a href="https://msdn.microsoft.com/en%20us/library/ayeb3ayc.aspx">https://msdn.microsoft.com/en us/library/ayeb3ayc.aspx</a> Some legacy source code may try to access the contents of an __m128 struct directly so the developer can use the SIMDVec as an alias for it. Any casting must be done manually by the developer, as you cannot cast or otherwise alias the base NEON data type for intrinsic operations.<br />
<br />
A bug was found with the <b>_mm_shuffle_ps</b> intrinsic. If the shuffle permutation was not one of the ones with a custom/unique implementation causing it to fall through to the default shuffle implementation it was failing to return the correct value. This is now fixed.<br />
<br />
A bug was found with the <b>_mm_cvtps_epi32</b> intrinsic. This converts floating point values to integers. It was not honoring the correct rounding mode. In SSE the default rounding mode when converting from float to int is to use 'round to even' otherwise known as '<a href="http://wiki.c2.com/?BankersRounding">bankers rounding</a>'. ARMv7 did not support this feature but ARMv8 does. As it stands today, this header file assumes ARMv8. If you are trying to target really old ARM devices, you may get a build error.<br />
<br />
Support for a number of new intrinsics was added, however, none of them yet have unit-tests to 100% confirm they are producing the correct results on NEON. These unit tests will be added as soon as possible.<br />
<br />
Here is the list of new instrinsics which have been added:<br />
<br />
<br />
<ul>
<li> <b>_mm_cvtss_f32</b> : extracts the lower order floating point value from the parameter</li>
<li> <b>_mm_add_ss</b> : adds the scalar single - precision floating point values of a and b</li>
<li> <b>_mm_div_ps</b> : Divides the four single - precision, floating - point values of a and b.</li>
<li> <b>_mm_div_ss</b> : Divides the scalar single - precision floating point value of a by b.</li>
<li> <b>_mm_sqrt_ss</b> : Computes the approximation of the square root of the scalar single - precision floating point value of in.</li>
<li><b>_mm_rsqrt_ps</b> : Computes the approximations of the reciprocal square roots of the four single - precision floating point values of in.</li>
<li><b>_mm_comilt_ss</b> : Compares the lower single - precision floating point scalar values of a and b using a less than operation</li>
<li><b>_mm_comigt_ss</b> : Compares the lower single - precision floating point scalar values of a and b using a greater than operation.</li>
<li><b>_mm_comile_ss</b> : Compares the lower single - precision floating point scalar values of a and b using a less than or equal operation.</li>
<li><b>_mm_comige_ss</b> : Compares the lower single - precision floating point scalar values of a and b using a greater than or equal operation.</li>
<li><b>_mm_comieq_ss</b> : Compares the lower single - precision floating point scalar values of a and b using an equality operation.</li>
<li><b>_mm_comineq_s</b> : Compares the lower single - precision floating point scalar values of a and b using an inequality operation</li>
<li><b>_mm_unpackhi_epi8</b> : Interleaves the upper 8 signed or unsigned 8 - bit integers in a with the upper 8 signed or unsigned 8 - bit integers in b.</li>
<li><b>_mm_unpackhi_epi16</b>: Interleaves the upper 4 signed or unsigned 16 - bit integers in a with the upper 4 signed or unsigned 16 - bit integers in b.</li>
</ul>
<br />
<br />John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com0tag:blogger.com,1999:blog-23430315.post-18279754640560477032015-09-04T09:51:00.000-07:002015-09-04T09:51:00.522-07:00PathCanonicalize and PathFindName functions<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6SULqfV0hpz1XYhYkhG8qxzyocpSkBuoXfR28PE3qDAYVOJci92tuEzi140vibTDxQ3bqy0Khx_cL0PmHmNxIcnbicZTXjHbDrVJPIhrnr6kg9IHK88tIU6jLdUPqEwon3F3Tqw/s1600/dumpster.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6SULqfV0hpz1XYhYkhG8qxzyocpSkBuoXfR28PE3qDAYVOJci92tuEzi140vibTDxQ3bqy0Khx_cL0PmHmNxIcnbicZTXjHbDrVJPIhrnr6kg9IHK88tIU6jLdUPqEwon3F3Tqw/s400/dumpster.jpg" width="353" /></a></div>
<br />
So, I just experienced a Google search fail. I inherited some code which made use of the Windows API calls 'PathCanonicalize' and 'PathFindName' functions. Since most of my work these days is multi-platform, I needed a version of these routines I could compile from source.<br />
<br />
I did a reasonable amount of Google searching and I couldn't find anything. Which was rather annoying. It's not that these routines are very difficult to write, I think it took me less than 30 minutes which included testing for edge cases. But that is 30 minutes I would rather not have wasted.<br />
<br />
So, I am posting my C implementation of these two routines here so that others don't have to go through this mild annoyance.<br />
<br />
There is one difference between my version and the Windows version of these routines, the Windows version only parses backslashes, where my routine is agnostic and treats backslashes or forwardslashes the same.<br />
<br />
<br />
<a name='more'></a><br />
<a href="https://drive.google.com/file/d/0BwdyTvSh6bUkLW04OGNRRnlxeDA/view?usp=sharing">Here is a link to the code snippet.</a>John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com0tag:blogger.com,1999:blog-23430315.post-33424849919743147372015-06-15T07:59:00.002-07:002015-06-15T07:59:59.815-07:00SSE2NEON.h moved to GitHub<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcp-OsE9rCJSfOBottQL70cUXfufD0ObKn_7GTpgUxVAmyBf3SnODJ1NxMQn5-khq3TG_Y3C0kBgPj_aKntIWCLNR9q3Bcvg3OG94lQv6ZdwZIznUu9u8XWjdduvBSgjL2otJVRg/s1600/IMG_20150614_160606.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcp-OsE9rCJSfOBottQL70cUXfufD0ObKn_7GTpgUxVAmyBf3SnODJ1NxMQn5-khq3TG_Y3C0kBgPj_aKntIWCLNR9q3Bcvg3OG94lQv6ZdwZIznUu9u8XWjdduvBSgjL2otJVRg/s640/IMG_20150614_160606.jpg" width="640" /></a></div>
<br />
I'm sure everyone has heard by now that Google is dropping support for their code hosting service called 'Google Code'. Since I spent years getting all of my various open source projects on there, it is now going to take me god knows how long to migrate them to GitHub. <br />
<br />
The one project that I have moved so far is '<a href="https://github.com/jratcliff63367/sse2neon/blob/master/SSE2NEON.h">SSE2NEON.h</a>'. This is a perfect example of exactly why I release some stuff open source. <a href="https://github.com/jratcliff63367/sse2neon/blob/master/SSE2NEON.h">SSE2NEON.h</a> is a header file which makes it easy to port SSE intrinsics to Neon (PC to Android).<br />
<br />
I wrote the original version and I only ported as many methods as I needed to get a particular project to work. I didn't attempt to get complete and full coverage of the entire instruction set. Then, a co-worker of mine at NVIDIA named Brandon Rowlett, who was also working on a porting project, added some additional support.<br />
<br />
However, the most gratifying feedback I got was when a developer I have never met downloaded the header file for use in his own porting project and, then, he added even more support. I cannot say that we have full coverage at this time, but clearly it can make porting a big pile of SSE code to NEON a whole lot easier.<br />
<br />
Thanks to all of the contributors who have helped maintain this piece of reference code!<br />
<br />
You can find the revised header file at its new official location on GitHub here:<br />
<br />
<a href="https://github.com/jratcliff63367/sse2neon/blob/master/SSE2NEON.h">https://github.com/jratcliff63367/sse2neon/blob/master/SSE2NEON.h</a>John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com0tag:blogger.com,1999:blog-23430315.post-42293880752682786362015-03-12T14:02:00.003-07:002015-03-12T16:25:10.426-07:00Aaarrgghhhhh... Google shuts down Google Code!<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-8z8HzuSmpa8/VQH97YE4A1I/AAAAAAAAcAA/ymHOHWFG8x4/s1600/AArrgghh!.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-8z8HzuSmpa8/VQH97YE4A1I/AAAAAAAAcAA/ymHOHWFG8x4/s1600/AArrgghh!.JPG" /></a></div>
<br />
<br />
This completely sucks! Google announced today that they are shutting down Google Code; which is where I host *all* of my open source projects, including the bitcoin blockchain parser!<br />
<br />
Their excuse for doing this is that too many people are spamming the free service. I have a simple solution, stop making it free!! I would gladly pay a reasonable annual fee to have my projects hosted there. Their site was simply awesome to use.<br />
<br />
Apparently the industry is moving towards GitHub. Sorry, but my preferred source control system for open source projects is SVN. It is super easy to use and set up.<br />
<br />
From my experience, GitHub is a honking pile of stinking shit which is incredibly difficult to both use and understand.<br />
<br />
Now, I'm willing to be convinced otherwise....but.. I haven't been impressed yet.<br />
<br />
Someone convince me why I shouldn't think GitHub is a confusing and difficult to use piece of shit? Seriously, I want to be convinced.<br />
<br />
Having to migrate all of my open source projects over to GitHub is going to be a major pain in the ass. <br />
<br />
Come on Google, make Google Code a paid service and take my damned money please!<br />
<br />
*** EDIT ***<br />
<br />
I have some friends who are telling me that I am being a whiny bitch and that Git is the greatest thing that has ever existed since the beginning of time. I remain highly skeptical, but I promise to at least give it a try.John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com4tag:blogger.com,1999:blog-23430315.post-29329544885256232832015-02-05T16:05:00.001-08:002015-02-06T11:34:22.365-08:00SSE2NEON.h : A porting guide and header file to convert SSE intrinsics to their ARM NEON equivalent<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjn9EA_zlRy823jzjA2tqj6UBu0YgdT9pv6K8e6M5AoNWwVanr5kUtjB7uBkmGcbRPbQ8lPJ2XDKoliYw-8dkb_erZYgHZUNm2LkV_tkJr5BV-r4hWVj-4i8jcoUV7l2onJ9_k7_A/s1600/neon.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjn9EA_zlRy823jzjA2tqj6UBu0YgdT9pv6K8e6M5AoNWwVanr5kUtjB7uBkmGcbRPbQ8lPJ2XDKoliYw-8dkb_erZYgHZUNm2LkV_tkJr5BV-r4hWVj-4i8jcoUV7l2onJ9_k7_A/s1600/neon.jpg" height="265" width="400" /></a></div>
<br />
For a work project, I am porting some PC code which makes extensive use of SSE intrinsics over to Android. Everyone will tell you that the best way to port SSE to NEON is simply to rewrite all of the code. Well, that is easier said than done. Especially when you are porting a lot of SSE code that you did not, personally, write. Especially if that SSE code is a pile of macro expansion on top of macro expansion. All of this macro expansion makes rewriting (or even understanding) the original code pretty challenging.<br />
<br />
Here are some observations I made:<br />
<br />
<br />
<ul>
<li>Online documentation for SSE intrinsics is less than ideal. <a href="https://msdn.microsoft.com/en-us/library/y0dh78ez%28v=vs.90%29.aspx">Microsoft's documentation</a> is ok, better than nothing, but could certainly have more detail. </li>
<li><a href="https://software.intel.com/sites/landingpage/IntrinsicsGuide/">Intel's documentation</a> for the SSE instruction set is much better, as it shows full inputs, outputs, and even has psuedocode for each instruction.</li>
<li>Online documentation for NEON intrinsics is really, really weak. The main resource I was using for NEON documentation is <a href="http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0491c/BEHHCHAE.html">from ARM here.</a> While it is certainly better than nothing, it is missing a lot of detail that you just kind of have to figure out for yourself.</li>
<li>Here is an <a href="http://community.arm.com/groups/tools/blog/2012/07/23/nevada-neon-visualization-for-arm-developers">article and link</a> to an interesting tool that lets you step through a visual simulator of ARM NEON instructions.</li>
<li>There doesn't appear to be one central reference that shows how to convert SSE to NEON; instead there are just a bunch of scattered posts on various programmer forums.</li>
</ul>
<div>
<br /></div>
<div>
Now, if you want to convert from NEON to SSE, there is a solution. On December 12, 2012 (if I recall correctly that was the date of the great Mayan prophecy of doom) an engineer with Intel, <a href="https://www.linkedin.com/pub/victoria-zhislina/0/903/143">Victoria Zhislina</a>, published the following <a href="https://software.intel.com/en-us/blogs/2012/12/12/from-arm-neon-to-intel-mmxsse-automatic-porting-solution-tips-and-tricks">excellent article and piece of source code</a>. This single piece of source remaps the NEON intrinsics API to functional equivalents for SSE. This is a fantastic resource and you can learn a lot from it. However, the API is hardly bijective, meaning there is no exact one to one mapping going both directions. </div>
<div>
<br /></div>
<div>
The NEON intrinsics are more powerful and general purpose than SSE, but they don't always map one to one.</div>
<div>
<br /></div>
<div>
For my specific task I needed to convert a subset of the SSE intrinsics to NEON; <b><i>but not the entire API!</i></b></div>
<div>
<b><i><br /></i></b></div>
<div>
I did exactly this, and the result is the following header file: <a href="https://code.google.com/p/sse2neon/source/browse/trunk/SSE2NEON.h">SSE2NEON.h</a></div>
<div>
<br /></div>
<div>
The idea is that for your Android application you can include this one header file, in place of:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"><i> #include "xmmintrin.h"</i></span><br />
<span style="font-family: Courier New, Courier, monospace;"><i> #include "emmintrin.h"</i></span><br />
<br /></div>
<div>
Then your code should just compile as normal with no changes. This did, in fact, work for my project. However, it is important to note, that I am only using a subset of the SSE intrinsics API.</div>
<div>
<br /></div>
<div>
To test that my port worked, I created a piece of code to 'unit test' each API call. What I did was write a C implementation of every routine and then compared it to the results of the corresponding intrinsics API call. I ran hundreds of random samples through every test and if at any point the C version and the intrinsics version didn't get the same result it will fire an assertion.</div>
<div>
<br /></div>
<div>
An important note. The routine to compute a reciprocol is an approximation on both the SSE and the NEON, therefore the unit test code looks for a match within an epsilon value rather than absolute.</div>
<div>
<br /></div>
<div>
Here is the <a href="https://code.google.com/p/sse2neon/source/browse/trunk/SSE2NEONTEST.cpp">source code to the unit-test method.</a></div>
<div>
<br /></div>
<div>
The entire open source project is <a href="https://code.google.com/p/sse2neon/">located here.</a> I have only provided a build configuration for Visual Studio. To test it with your own android project, just copy the source files over and add them. They have all been tested with my own Android builds.</div>
<div>
<br /></div>
<div>
My major motivation in writing this post and releasing this open source is so that no other person ever has to go through the hassle I went through trying to get this reference. I did finally get all of this to work, but only after bugging a bunch of my co-workers for help and digging through endless Google searches all over the web.</div>
<div>
<br /></div>
<div>
Also, as I have already mentioned, this implementation is incomplete. I only implemented as many SSE intrinsics as I needed to get my project to work. I don't have the time right now to try to add a NEON implementation for every single API call. </div>
<div>
<br /></div>
<div>
However, going forward, I would imagine that many other programmers are going to be faced with porting SSE code to NEON, so this could serve as a great reference. If you are doing an SSE to NEON port and want to use this source code, let me know, and I will add you as a contributor to the project. This way, over time, just like a good stone soup, we can eventually get a complete and fully vetted reference implementation everyone can share.</div>
<div>
<br /></div>
<div>
To help with Google Searches, I am going to list each API call I have ported to date, with a note about their status and NEON equivalent.</div>
<div>
<br /></div>
<div style="text-align: center;">
<u><b>Set and Get functions</b></u></div>
<div>
<u><b><br /></b></u></div>
<div>
<ul>
<li><b>_mm_setzero_si128</b> : maps directly to : <b>vdupq_n_s32</b>(0); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/ys7dw0kh(v=vs.100).aspx">Sets the 128-bit value to zero</a></li>
<li><b>_mm_setzero_ps</b> : maps directly to : <b>vdupq_n_f32</b>(0); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/tk1t2tbz(v=vs.100).aspx">Clears the four single-precision, floating-point values.</a></li>
<li><b>_mm_set1_ps</b> : maps directly to : <b>vdupq_n_f32</b>(w); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/2x1se8ha(v=vs.100).aspx">Sets the four single-precision, floating-point values to w.</a></li>
<li><b>_mm_set_ps</b> : maps to : <b>vld1q_f32</b>(data); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/afh0zf75(v=vs.100).aspx">Sets the four single-precision, floating-point values to the four inputs.</a></li>
<li><b>_mm_set1_epi32</b> : maps directly to : <b>vdupq_n_s32</b>(_i); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/h4xscxat(v=vs.100).aspx">Sets the 4 signed 32-bit integer values to i.</a></li>
<li><b>_mm_set_epi32</b> : maps to : <b>vld1q_s32</b>(data); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/019beekt(v=vs.100).aspx">Sets the 4 signed 32-bit integer values.</a></li>
<li><b>_mm_store_ps</b> : maps to : <b>vst1q_f32</b>(p,a); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/s3h4ay6y(v=vs.100).aspx">Stores four single-precision, floating-point values</a></li>
<li><b>_mm_load1_ps</b> : maps directly to : <b>vld1q_dup_f32</b>(p); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/5cdkf716(v=vs.100).aspx">Loads a single single-precision, floating-point value, copying it into all four words</a></li>
<li><b>_mm_load_ps</b> : maps directly to : <b>vld1q_f32</b>(p) : <a href="https://msdn.microsoft.com/en-us/library/vstudio/zzd50xxt(v=vs.100).aspx">Loads four single-precision, floating-point values.</a></li>
</ul>
</div>
<div style="text-align: center;">
<b><u>Logic and Binary Operations</u></b></div>
<div style="text-align: left;">
<b><u><br /></u></b></div>
<div style="text-align: left;">
<ul>
<li><b>_mm_andnot_ps </b>: maps (with argument swap) and cast to : <b>vbicq_s32</b>((__m128i)b,(__m128i)a); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/68h7wd02(v=vs.100).aspx">Computes the bitwise AND-NOT of the four single-precision, floating-point values of a and b.</a></li>
<li><b>_mm_andnot_si128</b> : maps (with argument swap) to : <b>vbicq_s32</b>(b,a); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/1beaceh8(v=vs.100).aspx">Computes the bitwise AND of the 128-bit value in b and the bitwise NOT of the 128-bit value in a.</a></li>
<li><b>_mm_and_si128</b> : maps directly to : <b>vandq_s32</b>(a,b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/6d1txsa8(v=vs.100).aspx">Computes the bitwise AND of the 128-bit value in a and the 128-bit value in b</a></li>
<li><b>_mm_and_ps</b> : maps directly with cast to : <b>vandq_s32</b>((__m128i)a,(__m128i)b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/73ck1xc5(v=vs.100).aspx">Computes the bitwise AND of the four single-precision, floating-point values of a and b.</a></li>
<li><b>_mm_or_ps</b> : maps directly with cast to : <b>vorrq_s32</b>((__m128i)a,(__m128i)b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/7ctdsyy0(v=vs.100).aspx">Computes the bitwise OR of the four single-precision, floating-point values of a and b.</a></li>
<li><b>_mm_or_si128</b> : maps directly to : <b>vorrq_s32</b>(a,b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/ew8ty0db(v=vs.100).aspx">Computes the bitwise OR of the 128-bit value in a and the 128-bit value in b.</a></li>
<li><b>_mm_movemask_ps</b> : does not map directly, may need a more efficient implementation. : <a href="https://msdn.microsoft.com/en-us/library/vstudio/4490ys29(v=vs.100).aspx">Creates a 4-bit mask from the most significant bits of the four single-precision, floating-point values.</a></li>
<li><b>_mm_shuffle_ps</b> : does not map directly, may need a more efficient implementation. : <a href="https://msdn.microsoft.com/en-us/library/vstudio/5f0858x0(v=vs.100).aspx">Selects four specific single-precision, floating-point values from a and b, based on the mask i.</a></li>
<li><b>_mm_slli_epi32</b> : maps directly to : <b>vshlq_n_s32</b>(a,b) : <a href="https://msdn.microsoft.com/en-us/library/z2k3bbtb%28v=vs.90%29.aspx">Shifts the 4 signed or unsigned 32-bit integers in a left by count bits while shifting in zeros.</a></li>
<li><b>_mm_movemask_epi8</b> : does ot map directly; may need a more efficient implementation. : <a href="https://msdn.microsoft.com/en-us/library/vstudio/s090c8fk(v=vs.100).aspx">Creates a 16-bit mask from the most significant bits of the 16 signed or unsigned 8-bit integers in a and zero extends the upper bits.</a></li>
</ul>
</div>
<div style="text-align: center;">
<b><u>Math Operations</u></b></div>
<div style="text-align: left;">
<b><u><br /></u></b></div>
<div style="text-align: left;">
<ul>
<li><b>_mm_sub_ps </b>: maps directly to : <b>vsubq_f32</b>(a,b); :<a href="https://msdn.microsoft.com/en-us/library/vstudio/1zad2k61(v=vs.100).aspx"> Subtracts the four single-precision, floating-point values of a and b</a>.</li>
<li><b>_mm_sub_epi32</b> : maps directly to : <b>vsubq_s32</b>(a,b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/fhh866h0(v=vs.100).aspx">Subtracts the 4 signed or unsigned 32-bit integers of b from the 4 signed or unsigned 32-bit integers of a.</a></li>
<li><b>_mm_add_ps</b> : maps directly to : <b>vaddq_f32</b>(a,b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/c9848chc(v=vs.100).aspx">Adds the four single-precision, floating-point values of a and b.</a></li>
<li><b>_mm_add_epi32</b> : maps directly to : <b>vaddq_s32</b>(a,b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/09xs4fkk(v=vs.100).aspx">Adds the 4 signed or unsigned 32-bit integers in a to the 4 signed or unsigned 32-bit integers in b.</a></li>
<li><b>_mm_mullo_epi16</b> : maps directly to : <b>vmulq_s16</b>((int16x8_t)a,(int16x8_t)b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/9ks1472s(v=vs.100).aspx">Multiplies the 8 signed or unsigned 16-bit integers from a by the 8 signed or unsigned 16-bit integers from b.</a></li>
<li><b>_mm_mul_ps</b> : maps directly to : <b>vmulq_f32</b>(a,b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/22kbk6t9(v=vs.100).aspx">Multiplies the four single-precision, floating-point values of a and b.</a></li>
<li><b>_mm_rcp_ps</b> : maps partially to : <b>vrecpeq_f32 </b>: Requires an extra approximation iteration to match SSE : <a href="https://msdn.microsoft.com/en-us/library/vstudio/796k1tty(v=vs.100).aspx">Computes the approximations of reciprocals of the four single-precision, floating-point values of a.</a></li>
<li><b>_mm_max_ps</b> : maps directly to : <b>vmaxq_f32</b>(a,b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/ff5d607a(v=vs.100).aspx">Computes the maximums of the four single-precision, floating-point values of a and b.</a></li>
<li><b>_mm_min_ps</b> : maps directly to : <b>vminq_f32</b>(a,b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/wh13kadz(v=vs.100).aspx">Computes the minima of the four single-precision, floating-point values of a and b.</a></li>
<li><b>_mm_min_epi16</b> : maps directly to : <b>vminq_s16</b>((int16x8_t)a,(int16x8_t)b); :<a href="https://msdn.microsoft.com/en-us/library/vstudio/6te997ew(v=vs.100).aspx"> Computes the pairwise minima of the 8 signed 16-bit integers from a and the 8 signed 16-bit integers from b.</a></li>
<li><b>_mm_mulhi_epi16</b> : maps indirectly to these two functions : <b>vqdmulhq_s16</b>((int16x8_t)a,(int16x8_t)b); and <b>vshrq_n_s16</b>(ret,1); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/59hddw1d(v=vs.100).aspx%20inline%20__m128i%20_mm_mulhi_epi16%20(__m128i%20a,%20__m128i%20b)">Multiplies the 8 signed 16-bit integers from a by the 8 signed 16-bit integers from b.</a></li>
</ul>
</div>
<div style="text-align: center;">
<b><u>Compare Operations</u></b></div>
<div style="text-align: left;">
<b><u><br /></u></b></div>
<div style="text-align: left;">
<ul>
<li><b>_mm_cmplt_ps</b> : maps directly to : <b>vcltq_f32</b>(a,b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/f330yhc8(v=vs.100).aspx">Compares for less than</a></li>
<li><b>_mm_cmpgt_ps</b> : maps directly to : <b>vcgtq_f32</b>(a,b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/11dy102s(v=vs.100).aspx">Compares for greater than.</a></li>
<li><b>_mm_cmpge_ps</b> : maps directly to : <b>vcgeq_f32</b>(a,b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/fs813y2t(v=vs.100).aspx">Compares for greater than or equal</a></li>
<li><b>_mm_cmple_ps</b> : maps directly to : <b>vcleq_f32</b>(a,b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/1s75w83z(v=vs.100).aspx">Compares for less than or equal.</a></li>
<li><b>_mm_cmpeq_ps</b> : maps directly to : <b>vceqq_f32</b>(a,b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/36aectz5(v=vs.100).aspx">Compares for equality</a>.</li>
<li><b>_mm_cmplt_epi32</b> : maps directly to : <b>vcltq_s32</b>(a,b); :<a href="https://msdn.microsoft.com/en-us/library/vstudio/4ak0bf5d(v=vs.100).aspx"> Compares the 4 signed 32-bit integers in a and the 4 signed 32-bit integers in b for less than.</a></li>
<li><b>_mm_cmpgt_epi32</b> : maps directly to : <b>vcgtq_s32</b>(a,b); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/1s9f2z0y(v=vs.100).aspx">Compares the 4 signed 32-bit integers in a and the 4 signed 32-bit integers in b for greater than.</a></li>
</ul>
</div>
<div style="text-align: center;">
<b><u>Conversion Operations</u></b></div>
<div style="text-align: left;">
<ul>
<li><b>_mm_cvttps_epi32</b> : maps directly to : <b>vcvtq_s32_f32</b>(a); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/1h005y6x(v=vs.100).aspx">Converts the four single-precision, floating-point values of a to signed 32-bit integer values using truncate.</a></li>
<li><b>_mm_cvtepi32_ps</b> : maps directly to : <b>vcvtq_f32_s32</b>(a); : <a href="https://msdn.microsoft.com/en-us/library/vstudio/36bwxcx5(v=vs.100).aspx">Converts the four signed 32-bit integer values of a to single-precision, floating-point values</a></li>
<li><b>_mm_cvtps_epi32</b> : maps directly to : <b>vcvtaq_s32_f32</b>(a); <a href="https://msdn.microsoft.com/en-us/library/vstudio/xdc42k5e(v=vs.100).aspx">for ARMV8 and above, otherwise it takes several instructions</a></li>
</ul>
</div>
John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com5tag:blogger.com,1999:blog-23430315.post-61377723373061998272014-07-05T09:35:00.002-07:002014-07-15T14:44:48.766-07:00Everyone Deserves a Good Origin Story<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div style="line-height: normal;">
<br /></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="color: black; font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"><img height="231px;" src="https://lh3.googleusercontent.com/3uaYzWyQAFEhtA5CMLfvjXI9uZ8NC9uL08ysS7lW_jBMkR9qtpCqgeYIPE8HFukKhQfjjce_eELU4N__JSwBBH0lAqRUUZvrz4hwYvd6fPN4OpEmKiAw51m5-Vtdb5mfrQ" style="-webkit-transform: rotate(0rad); border: none;" width="125px;" /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
<span style="color: black; font-family: Arial; font-size: 15px; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">(The Texas Instruments TI-55 programmable calculator which came out in 1977)</span></div>
<div style="line-height: normal;">
<br /></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;"> </span></div>
</div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have been feeling a bit nostalgic lately as I started thinking about how I got into the game industry. The other day I stumbled across an old scrapbook which contained clippings from when my first game was published. This inspired me to write the following elegiac post.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The first time I ever got my hands on a device that was programmable was in 1979 at my high school graduation. My parents bought me a </span><a href="http://en.wikipedia.org/wiki/TI-55" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">TI-55</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> programmable calculator which supported a grand total of 32 programming steps. I was completely enthralled by the power of being able to program into the device my own equations and functions. My older brother was working towards an electrical engineering degree at the time and, when he saw the things I was making this calculator do, he encouraged me to pursue a degree in computer science instead of my original plan to follow his degree path.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizKzv5FwF99N11hbpgJA1iTwNGJRiE_MkrVTBXQG0YWPdCkzTt2Bs3Fa-3ofTit2mO_uAy6RZxTYaMipjnnNTgqifoh4QyGx4R1_krimSerbMcbEK4XwzxpHhC2ONS7keDdaANYQ/s1600/apple-ii.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizKzv5FwF99N11hbpgJA1iTwNGJRiE_MkrVTBXQG0YWPdCkzTt2Bs3Fa-3ofTit2mO_uAy6RZxTYaMipjnnNTgqifoh4QyGx4R1_krimSerbMcbEK4XwzxpHhC2ONS7keDdaANYQ/s1600/apple-ii.jpg" height="210" width="320" /></a></div>
<br /></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">When I first started college I got my hands on an original </span><a href="http://en.wikipedia.org/wiki/Apple_II_series" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Apple II</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> computer which, I must admit, was a giant leap forward from the TI-55 calculator. I quickly went to work mastering this incredible machine that Steve Wozniak built. During college I was already creating educational software and, from that, I got my first ever paid programming gig writing a report card system for a local high-school. This turned out to be an early life lesson since, long after I pocketed the $450 they paid me to write an entire system which computed the GPA for every student, tracked all progress, and printed out report cards, they kept expecting me to maintain it for free for years to come. This life lesson would repeat itself over and over again throughout my career whenever I tried to market a piece of middleware.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">After college I got my first job writing educational software for a forward thinking company called Milliken Publishing. During the early years Apple had a strategy of getting their computers into every school in the country, offering them for free or at deep discounts. This created one of the first software markets as schools suddenly had all of these computers and wanted to make use of them in the classroom. Milliken was innovative enough to hire computer programmers to create a wide catalog of educational software that was curriculum based. They designed it so that a school which bought their books and supplemental materials could also get the same curriculum delivered on the Apple II computer as well. Unlike other companies who were just turning basic math drills into simple games, Milliken produced teacher developed curriculum and used innovative teaching methods. Even though we delivered a lot of solid curriculum, we also went out of our way to make the software fun and engaging for the students.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpyalZ-JmueE_BRv0jwK-NrBDbnZjZ2-vRS_DbEX-b88fRsgNRURvmN6-UQe2os_rWycRG8CZRe4qg7YtdoGVDLcyDSseXBU8panQ_nMSJcroSg8JqmrDMFlbDMjcsbcRNVIYa2Q/s1600/download+(1).jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpyalZ-JmueE_BRv0jwK-NrBDbnZjZ2-vRS_DbEX-b88fRsgNRURvmN6-UQe2os_rWycRG8CZRe4qg7YtdoGVDLcyDSseXBU8panQ_nMSJcroSg8JqmrDMFlbDMjcsbcRNVIYa2Q/s1600/download+(1).jpg" /></a></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">At the time, even though I wasn't directly creating computer games, I was certainly following the industry. In the summer of 1979 </span><a href="http://en.wikipedia.org/wiki/Richard_Garriott" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Richard Garriot</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> was working at a </span><a href="http://en.wikipedia.org/wiki/ComputerLand" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">ComputerLand</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> store when he had his first encounter with the Apple II computer as well. Just a short while before I was busy writing report card software for New Haven High School Richard was creating one of the first computer games in history; </span><a href="http://en.wikipedia.org/wiki/Akalabeth" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Akalabeth</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. He took his game, stored on a cassette tape, put it in a ziploc bag, and sold it at the local Apple store. Richard only sold about a dozen copies at the store but one of those copies made it to a company called </span><a href="http://en.wikipedia.org/wiki/California_Pacific_Computer_Company" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">California Pacific</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> who signed a deal with him. Ultimately the game sold 30,000 copies with Richard receiving about $5 apiece. At the time this was considered a blockbuster mega-hit computer game. Richard went on to create the </span><a href="http://en.wikipedia.org/wiki/Ultima_(series)" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Ultima </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">series and has had an amazing career and a life of adventure. Today he is probably just as famous for becoming an astronaut and being part of the first father-son team to make it into space.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As my career as a software engineer continued I, like so many others, desperately wanted to break into the game industry. I knew I could make a game, but I just didn't know how to get started. I had created a piece of graphics technology to render a 3d height field as a solid shaded visualization and I thought I could use it to create some kind of first person shooter. With this idea in mind I created a game prototype and was ready to send it off to Electronic Arts.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwi9y2XsAXFkp_dodEZ3QU2kC2KQkRcJJryXvYRkfoN3QJ-QLFsvcld2RYYd5YVjUwADFPSV8dNyLMl69A6DwFVTjOvFD2_TuJccLBKXVsm0LnTdUP2H-rbqcGvQgyS8WiB2I-Og/s1600/download.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwi9y2XsAXFkp_dodEZ3QU2kC2KQkRcJJryXvYRkfoN3QJ-QLFsvcld2RYYd5YVjUwADFPSV8dNyLMl69A6DwFVTjOvFD2_TuJccLBKXVsm0LnTdUP2H-rbqcGvQgyS8WiB2I-Og/s1600/download.jpg" /></a></div>
<b style="font-weight: normal;"><br /></b>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://en.wikipedia.org/wiki/Electronic_arts" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Electronic Arts</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> was established by </span><a href="http://en.wikipedia.org/wiki/Trip_Hawkins" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Trip Hawkins</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> on May 28, 1982. Trip had a lot of great ideas about how computer games should be created and marketed that are almost entirely responsible for how the industry evolved over the last three decades. Trip wanted to co-opt the movie industry studio model and apply it to video games. His plan was to treat game developers as ‘creative artists’ who would work against advances on royalties while the studio would market and distribute the finished product. All of the job titles which exist today in the game industry, that of writer, producer, artist, production assistant, arise from Trip Hawkin’s original vision.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Trip’s vision of the company initially was also a pretty good business model. When Electronic Arts initially started, the company itself didn't actually create video games, or relatively few of them. Only over time did they move more and more game development in-house. During the early days hundreds of geeky kids working out of their parents homes would ‘pitch’ ideas to the ‘studio’ and try to get a production deal. It is important to remember that in these early days an entire computer game could be created, start to finish, by just a handful of people. Electronic Arts used a shotgun approach at the time, signing up a whole bunch of young people to try to create their games at very low cost. They offered advances on royalties which were only paid out as milestone deliveries were met. They could cancel the project at any time to mitigate risk if they felt it wasn't going well. Once a game was finished they would decide how much money to throw at it for marketing. If they didn't believe in the project, they would not put much money behind it at all. If they thought it could be a hit then they might put a stronger effort into print advertising. This was pre-internet days, so video games were primarily promoted through the dozens of dedicated gaming magazines which covered newsstands at the time. If any of the games were a big hit, then Electronic Arts would make enough money off of it that it could easily offset the costs of a few that never really went anywhere.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In 1987 I desperately wanted to create a </span><a href="http://en.wikipedia.org/wiki/First-person_shooter" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">first-person-shooter</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. This wasn't even really a genre at the time. Pretty much the only 3d games back then were flight simulators and one or two tank simulators, some of which were still using wireframe graphics. I had created a visualization technique which allowed me to render a 3d height field as a solid shaded terrain covered by rotoscoped sprite images for trees and players. It ran at an interactive frame-rate on a PC and, at the time, I thought it was pretty cool.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I began by cold-calling Electronic Arts asking to speak to someone about submitting my game idea. I eventually got to speak to a nice guy named Jon Manley who said he would send me a product idea </span><a href="https://drive.google.com/file/d/0BwdyTvSh6bUkcHA4SjU3X25VMjQ/edit?usp=sharing" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">submission packet</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> in the mail which I received shortly thereafter. When I got the packet it included the following </span><a href="https://drive.google.com/file/d/0BwdyTvSh6bUkajJmRHRxY2F4OFE/edit?usp=sharing" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">intimidating advice</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“Is this product pretty good? Very good? One of the best 100 products on the market? If your answer of to any of these questions is ‘yes’, don’t send it yet. The personal computer software market is so crowded that the only titles that have any real chance of breaking in are those that can be considered awesome, earth-shaking, world-class. Is this product unique? With the hundreds of submissions we receive, you can imagine how many times we see virtually the same product go by. The world is not clamoring for another graphics/text adventure game, climbing game, maze game and any number of other genres. To get the market’s attention-and ours--you have to be different.”</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Fast forward to 2014 where we see the same exact games getting recycled over, and over, and over, again, and you can read whatever irony into this that you want. Clearly somewhere on this journey we lost our way designing video games. To understand why, I can simply cut to the chase and explain exactly what happened. Video games became mainstream mass-market products. While in the early days Electronic Arts could fund twenty game projects built by small teams and be satisfied if just one or two of them hit, that isn’t the case any more. Today a single game has a budget larger than most Hollywood movies and massive teams of often hundreds of people working on it for years. There is almost zero margin for error. Back in 1984 if Electronic Arts had a game that sold 100,000 copies this was considered a monster mega-hit. Today, if a game was projected to sell 3 million units but only sells 2 million, well, you can expect to hear of yet one more round of layoffs in the game industry.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdlSW_QDxD9LuYRguj8QxWEiroxm6wvWlKYqe2xwEKpyYLokc59SZy6KkcwwPuT-a2B7HbMbuf2HItUSUBoMtKJP1lm0SxUoqrnhknWYUNDu3bkL7DHYyUUiJc-dNvFGxwt66UbA/s1600/ElectronicArtsSubmissionPage1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdlSW_QDxD9LuYRguj8QxWEiroxm6wvWlKYqe2xwEKpyYLokc59SZy6KkcwwPuT-a2B7HbMbuf2HItUSUBoMtKJP1lm0SxUoqrnhknWYUNDu3bkL7DHYyUUiJc-dNvFGxwt66UbA/s1600/ElectronicArtsSubmissionPage1.jpg" height="320" width="292" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Cover Page from the Electronic Arts Submission Packet</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgatlCYmILlS4SQy6fEHUmAEMXtMiHrJsL6B8MJmKWEgrRXLNEABtnIB899RxSm1zV-UxzaKN2CbFBiNK3l7BlQDYG0jmXO6rVnBAk78frFZ3fNvWc4GWarFWoatqDOzV2O3Uwa7Q/s1600/ElectronicArtsSubmissionPage2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgatlCYmILlS4SQy6fEHUmAEMXtMiHrJsL6B8MJmKWEgrRXLNEABtnIB899RxSm1zV-UxzaKN2CbFBiNK3l7BlQDYG0jmXO6rVnBAk78frFZ3fNvWc4GWarFWoatqDOzV2O3Uwa7Q/s1600/ElectronicArtsSubmissionPage2.jpg" height="320" width="225" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The Electronic Arts product submission guidelines page</td></tr>
</tbody></table>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: italic; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<br />
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">What started as a place where innovative ideas could be thrown out to a largely hobbyist market became a mainstream mass market and with all of the restrictions that this implies. Fortunately, due to the recent development of the mobile platform and certain break-out games like ‘Minecraft’, every now and then the industry is reminded that no matter how fancy your graphics are, no matter how much money you spend on production, at the end of the day it should be about new and exciting game design.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Getting back to my own story, in 1987 I sent in my, quite frankly, somewhat embarrassing product submission into Electronic Arts. It included a fully functional interactive demo which showcased the 3d visualization technique that I was proposing and the following </span><a href="https://drive.google.com/file/d/0BwdyTvSh6bUkbWRTZ3NoOUt0emc/edit?usp=sharing" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">cover page</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. Yes, I actually submitted this to Electronic Arts and, yes, I drew that ridiculous cover art myself. I will at least take some credit for having the vision at the time of wanting to create a first person shooter with a capture the flag element which later became a staple of the game industry.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7dRlMvDwrL0TdBJ7XMRH7t8kEqZwDmi1m13y1bcOJD2eLr0Ve_CTa_nzMBrOXdJxkPMKurcY_cFwTRMLwSWptlFhvxsho_bIgj9-ZRogceBIH-T9CbPq-K9wMw1-24oDqQ1gZ6A/s1600/Gotcha.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7dRlMvDwrL0TdBJ7XMRH7t8kEqZwDmi1m13y1bcOJD2eLr0Ve_CTa_nzMBrOXdJxkPMKurcY_cFwTRMLwSWptlFhvxsho_bIgj9-ZRogceBIH-T9CbPq-K9wMw1-24oDqQ1gZ6A/s1600/Gotcha.jpg" height="320" width="215" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The actual embarrassing cover page of my product submission to Electronic Arts</td></tr>
</tbody></table>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here is what happened next. About a week after I sent that off I got a phone call from an ‘associate producer’ at Electronic Arts by the name of </span><a href="http://www.mobygames.com/developer/sheet/view/developerId,511/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Paul Grace</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. To say I was excited would be an understatement; getting a call from someone, anyone, at Electronic Arts about my game submission was the biggest thing that had ever happened to me in my entire life. Paul was blunt on the phone. He made it immediately clear that he thought my game idea was lame and his first question was whether or not I was married to the ideal, with a tone in his voice that the call would soon be over if I said I was. What Paul was calling about was really two things, first he liked the 3d rendering technology and second he liked the fact that I could just program a game by myself. Even though Paul was just an associate producer at the time, what he really wanted to do was design and produce an entire game. At the time </span><a href="http://en.wikipedia.org/wiki/Tom_Clancy" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Tom Clancy</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> had just published ‘Hunt for Red October’ and submarines were all the rage. Paul wanted to make a submarine game and somehow he saw in my programming a way that he might possibly be able to do that. When he asked me, “How do you feel about working on a submarine game?” my eager and immediate response was, “If I can do a game with Electronic Arts I’ll do anything you want!” And, thus, was a partnership formed. I have asked around and no one at Electronic Arts can remember anyone else ever getting a game contract by sending in a cold submission to the mail-room before. I may not be the only one, but it was certainly rare.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Paul helped get my contract through the system so that he could use me to create the game that he wanted. We worked long and hard and the fact that I was married, had two kids, and a full-time job, meant that it took way longer to create this game than either of us were happy with. I worked on this game in the evenings and weekends, doing pretty much 100% of the programming myself while Paul hooked me up with a sound designer and artist to get the rest of the content for the game. Paul did almost all of the game design and created the missions.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The game used my 3d visualization technique to render ships, shore, and explosions for when the periscope was up and a simulated underwater contour view when you were below surface. Since sound is such a central part of all submarine stories I had to try to get sound and music to work in the game, something which was not at all easy to do since few people even had a sound card in their computer and there was no standardized way to use them. </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">During the course of the development of the game 16 color VGA graphics evolved into 256 color MCGA graphics. Most people have no idea how much it changed the computer gaming world when you could finally use 256 entire colors to display images. The difference in quality was astonishing, it was as if everything went from a cartoon to photo-realistic over-night. Simply due to this coincidental timing my game became one of the first products to take full advantage of 256 color graphics throughout all of the artwork. The game also took advantage of true digital audio on as wide a range of sound cards possible while, at the same time, even managed to produce crude digital sound through the PC speaker (for which I was threatened with patent violation by a patent troll even back then, but that is a story for another day.). Another unique feature at the time was that the game had multiplayer capabilities, one of the few that would let two players go against each other head to head via a crude low-speed modem connection. I remember having my modem dial and connect to Paul’s modem in San Mateo, hearing the screech resolve, and then the two of us fighting out imaginary submarine battles thousands of miles apart. It was a truly amazing experience at the time.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Once the game was finally done Paul faced a new challenge. Even though it had taken me seemingly forever to finish the game because, as I said, I still had a full time job, he was able to keep the project alive because it was relatively low cost to EA and it was, of course, his personal pet project. Now that the game was finally ready to ship the new problem we faced is that Electronic Arts didn't have any interest in putting a lot of money behind the marketing effort. What happened next is a little bit of gaming history. At that time there were a number of companies making a lot of money selling floppy disks. Floppy disks were the primary way that people who owned computers back then would store their data and games. One of the largest manufacturers of floppy disks at the time was Maxell and they had approached Electronic Arts with this crazy idea of putting a fully playable computer game on an extra free 11th floppy disk in each box. Electronic Arts marketing talked to a number of teams about this idea but no one was really very interested. It seemed completely insane to give away a fully playable version of a game. Why would anyone do that? Why would anyone buy the game if they could play it for free? Now that my game was ready to release, and it didn't have a lot of money invested in it, they decided to let me be the guinea pig. We made a full version of the game that you could play completely but, of course, only a single ‘game level’ or ‘mission’ was provided.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh42JTn2123WYCS1DqL5ghokG7FYlWEGAfYCJeLdv53c7e4VKbgOTbq2frvrZyZ0sBj2ElHlaDS4YTGpE7lxT0evh9GOSjiqf09yTeibIQIcpm78dM6yb13T2RQPJncqF0L70CGiw/s1600/MaxwellFullPageAd.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh42JTn2123WYCS1DqL5ghokG7FYlWEGAfYCJeLdv53c7e4VKbgOTbq2frvrZyZ0sBj2ElHlaDS4YTGpE7lxT0evh9GOSjiqf09yTeibIQIcpm78dM6yb13T2RQPJncqF0L70CGiw/s1600/MaxwellFullPageAd.jpg" height="320" width="224" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">This was the promotional image used by Maxwell and plastered all over Comdex and even on Las Vegas billboards</td></tr>
</tbody></table>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">At that years COMDEX there were actually </span><a href="https://drive.google.com/#folders/0BwdyTvSh6bUkcFBuT1h4Yl80N3c" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">billboards</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> in Las Vega with our game splashed across it! Over the coming months Maxell sold over a million boxes of floppy disks, each one containing a fully playable version of our game! This soon led to my first game, ‘</span><a href="http://en.wikipedia.org/wiki/688_Attack_Sub" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">688 Attack Sub</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">’, to become #1 on the bestseller list for several months and to sell over 150,000 copies which, at the time, was considered a huge hit. Later I ported the game to the Commodore Amiga and someone else ported it to the Sega Genesis system. </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">After this the entire game industry was convinced that a free playable demo was an important marketing tool and I’m proud to have been the first one to be subjected to this marketing experiment. When I was reviewing my old materials to write this story I came across </span><a href="https://drive.google.com/#folders/0BwdyTvSh6bUkcFBuT1h4Yl80N3c" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">this letter</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> I received from Dave Luoto which brought back a lot of fond memories.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPZAwSAilrykVO9DcivH7h82OM2ICavHOrrcOMzSfG7utPkWMkbyo8YyWzx5eJoj0cYygRtzenlGxZAc1ewwAhzlBsT9iMNJnni3KQ-OJisXlazwpt1FLVxO4FdU56gHzphz6dRA/s1600/MaxwellInsertPage1.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPZAwSAilrykVO9DcivH7h82OM2ICavHOrrcOMzSfG7utPkWMkbyo8YyWzx5eJoj0cYygRtzenlGxZAc1ewwAhzlBsT9iMNJnni3KQ-OJisXlazwpt1FLVxO4FdU56gHzphz6dRA/s1600/MaxwellInsertPage1.jpg" height="165" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The first page of the Maxell insert</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbq-z-TQBThTOm0vMTHnNn09upwMNn8-LNBiqA8ObXBmSffGTw47nxHUJg5CmVDKbdkdp3fTm1Rlan1MGzo0TaogeoDy3dY7fq5-0Ht2QdZvTS7nA-D9cS1HeZ6nHxs-hZ_Uh6gQ/s1600/MaxwellInsertPage2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbq-z-TQBThTOm0vMTHnNn09upwMNn8-LNBiqA8ObXBmSffGTw47nxHUJg5CmVDKbdkdp3fTm1Rlan1MGzo0TaogeoDy3dY7fq5-0Ht2QdZvTS7nA-D9cS1HeZ6nHxs-hZ_Uh6gQ/s1600/MaxwellInsertPage2.jpg" height="165" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The second page of the Maxwell insert</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdvq9mfS8ANmOXPz0zvJ2odPGCxzeX8YgYV0l1q5ESgPS78f88_aK5GJzCtzDgzW_SEESl2Z6sCBKBZWY_aTltU9gJ1bi5PPMI3iYllhV2sClItX4BCFYWUFi5l39qKb8GUb5VnQ/s1600/FrontCover688.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdvq9mfS8ANmOXPz0zvJ2odPGCxzeX8YgYV0l1q5ESgPS78f88_aK5GJzCtzDgzW_SEESl2Z6sCBKBZWY_aTltU9gJ1bi5PPMI3iYllhV2sClItX4BCFYWUFi5l39qKb8GUb5VnQ/s1600/FrontCover688.jpg" height="320" width="244" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The front cover of the '688 Attack Sub' box. Note that my name is listed as if I created the whole thing myself. </td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUDDRZR7aCSxXNX69XqJExzda4_fEjjq-zvN-qeoNdubfXM_Q8zsk-L2BUwAoISwNmDmrcgTuwdPeC-VTV_RFiil0UyHDMIg_uCkd8GRbeha8WC5vEKrzeJ8U7fGLYuMoDwebCjA/s1600/InsideCover688.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhUDDRZR7aCSxXNX69XqJExzda4_fEjjq-zvN-qeoNdubfXM_Q8zsk-L2BUwAoISwNmDmrcgTuwdPeC-VTV_RFiil0UyHDMIg_uCkd8GRbeha8WC5vEKrzeJ8U7fGLYuMoDwebCjA/s1600/InsideCover688.jpg" height="320" width="208" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The inside flap of the box which includes a personal bio</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyK8b7lg9tlO6rL2bEsVkCD2etC-G7fv2OLpqNqANyeixhQi1JneMZAayWZYgZIHFKiEc4pz6wZFo1VUACNseobt41gx4AeDji19xnNhVGvX_dny_iK0vWraWBXBTPl4tkwvWKqQ/s1600/BackCover688.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyK8b7lg9tlO6rL2bEsVkCD2etC-G7fv2OLpqNqANyeixhQi1JneMZAayWZYgZIHFKiEc4pz6wZFo1VUACNseobt41gx4AeDji19xnNhVGvX_dny_iK0vWraWBXBTPl4tkwvWKqQ/s1600/BackCover688.jpg" height="320" width="246" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The back of the box</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbhQA_it7ToGIcrX6cZFLMlwUO7mTRTatBalGoWxn50cQBpQbIGTwJW15ojMqcXVrfqAhEf2ObQZvUpKxdua_rzX8R4MJbkKroJVYEuko3_ZtMbL3C6VWRprsBmKobISNdmHo6iw/s1600/Floppy688.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbhQA_it7ToGIcrX6cZFLMlwUO7mTRTatBalGoWxn50cQBpQbIGTwJW15ojMqcXVrfqAhEf2ObQZvUpKxdua_rzX8R4MJbkKroJVYEuko3_ZtMbL3C6VWRprsBmKobISNdmHo6iw/s1600/Floppy688.jpg" height="304" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The actual 5 1/4" floppy disk that contained the game</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAkyrw9o5Tb_7-cPk1Zh8f4FmIrrzvjJIVy5QX73N44-_cuyyejLG2u3CKX5qPfkTQow8PsL18I_Q2v2xwmwV98-1C2RhiYRsL_cepT7ilLOrt5oE_A8f7acnoyNQaDm44P6IwGw/s1600/download+%25282%2529.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAkyrw9o5Tb_7-cPk1Zh8f4FmIrrzvjJIVy5QX73N44-_cuyyejLG2u3CKX5qPfkTQow8PsL18I_Q2v2xwmwV98-1C2RhiYRsL_cepT7ilLOrt5oE_A8f7acnoyNQaDm44P6IwGw/s1600/download+%25282%2529.jpg" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">I used photographs of Electronic Arts employees as ship officers in the game. Here Randy Breen pops up as the weapons officer politely informing the caption that he failed to select a weapon to launch.</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt; text-align: center;">
</div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2eDNEAX7gT7gnR0D1yBlzQDO4d5xVaDoEnoh2Mq4PfD53idg7gDqWy7ycnXEqfgDfZJOYAjVnp7c3ltBuX4l_4RnuB2cotcDAwH2e3KE6y5QZJrUc6F-XHckokYFbb8jAQ7VO2w/s1600/ElectronicArtsLetter.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2eDNEAX7gT7gnR0D1yBlzQDO4d5xVaDoEnoh2Mq4PfD53idg7gDqWy7ycnXEqfgDfZJOYAjVnp7c3ltBuX4l_4RnuB2cotcDAwH2e3KE6y5QZJrUc6F-XHckokYFbb8jAQ7VO2w/s1600/ElectronicArtsLetter.jpg" height="320" width="241" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">The letter I received from David Luoto upon the games completion</td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgw33DwD_csxoAeX6OyAaRRrH-k069ny3QIMeRaFlULaXsUxA8rjQIh_I0Cq67sDxi0xf5k_fz36IBuo3EKfdIQWiXdcYWpbTcxOg4UbLqTlnGmWV3AIIG7oyuFEJ85C36yY8kImw/s1600/Adlib.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgw33DwD_csxoAeX6OyAaRRrH-k069ny3QIMeRaFlULaXsUxA8rjQIh_I0Cq67sDxi0xf5k_fz36IBuo3EKfdIQWiXdcYWpbTcxOg4UbLqTlnGmWV3AIIG7oyuFEJ85C36yY8kImw/s1600/Adlib.jpg" height="320" width="231" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A promotional advertisement from AdLib, one of the first sound cards ever created, and supported by my game</td></tr>
</tbody></table>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have had a long career in the game industry with various levels of success on products, but always trying and pushing forward. After ‘688 Attack Sub’ we just turned right around and did another submarine game called ‘SSN-21 Seawolf’. Here is when I probably made the biggest mistake of my career. What I should have done was quit my ‘day job’ and become more serious about this whole games thing. I did not. For some reason I was always afraid to. With a wife and kids and a mortgage, I felt like I needed the security of a steady paycheck and I was not willing to take what I perceived as such a big risk. So during the entire development of ‘SSN-21 Seawolf’ I still maintained a full time job and this was really a disaster. The game did finally get finished and ship, but I was not ultimately very happy with it and it cost me a lot of time with my children that I can never get back. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I did improve the rendering technology and I definitely enhanced the audio substantially. The game did well commercially nonetheless and, as before, Paul did all of the game design and marshaled it through the Electronic Arts internal politics keeping me largely insulated from most of it. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">After ‘</span><a href="http://en.wikipedia.org/wiki/SSN-21_Seawolf" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">SSN-21 Seawolf</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">’ was complete, Paul finally was able to get me a contract for the project I wanted to do more than anything else, which was a first person shooter with a great deal of strategy. This time I did quit my day-job and took the plunge to make this my full-time effort. However, I made yet another mistake, While my friends and colleagues who were following a similar path went out and created their own game companies, hiring more and more people, I still just kept coding entire games by myself in my basement. I would subcontract out art and audio design, but I was still the only coder. A few years later when I would hear about my friends selling their companies for millions of dollars while I was still toiling away all by myself I was certainly a bit jealous. The fact of the matter is that I am just a coder at heart and I had no real interest in the process of running an entire business.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">After my next game shipped, called </span><a href="http://www.gamespot.com/reviews/scarab-review/1900-2532901/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">‘Scarab’ </span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">from Electronic Arts and it was a complete and total failure in the marketplace, it was finally time to move on. Even though ‘Scarab’ was a commercial failure it is by far my favorite game that I have ever been involved in. I did almost all of the game design and it represented a genre of game that I wish there was more of, a first person strategy game.</span></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Since then I have worked on a number of studio projects with ever increasingly large teams. I was the lead programmer on ‘Cyberstrike 2’ for 989 Studios and ‘Planetside’ published by Sony Online Entertainment.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Today I am a principal engineer at NVIDIA corporation where I work on various GameWorks projects which allows me to contribute technology that ends up in hundreds of games throughout the industry. It is by far, by leaps and bounds, the greatest job I have ever had in my life.</span></div>
John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com5tag:blogger.com,1999:blog-23430315.post-71580578016624081922014-05-16T14:08:00.001-07:002014-05-16T15:29:49.779-07:00RenderDebug and DebugView : A DLL Plugin and Viewer Application to perform Remote Debug Visualization on Windows<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqCCAnUjRPwrUFs_-lEcf5t1EDvviz1IxdkjrHHX9-GJzTz14bPiqdm2MaeVSEejwaB9aRrksCaEB-jAQqKJxaovulnUpWFOXL0k_eTIdOJBgMHwd951PQuoXDofnit3raHBiVWA/s1600/renderdebug.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhqCCAnUjRPwrUFs_-lEcf5t1EDvviz1IxdkjrHHX9-GJzTz14bPiqdm2MaeVSEejwaB9aRrksCaEB-jAQqKJxaovulnUpWFOXL0k_eTIdOJBgMHwd951PQuoXDofnit3raHBiVWA/s1600/renderdebug.png" height="528" width="640" /></a></div>
<br />
<h2 style="text-align: center;">
TLDR </h2>
<h2 style="text-align: center;">
How to quickly Run Bitcoin BlockChain Historical Data Visualization</h2>
<br />
<i><b>TLDR;</b> If you don't want to read this whole article and just want to download and run the visualization tool with the bitcoin blockchain historic data, here is what you should do. If you are a programmer and comfortable using SVN, then just sync to the SVN depot here:</i><br />
<i><br /></i>
<a href="https://code.google.com/p/renderdebug/"><i>renderdebug Google Code Project</i></a><br />
<i><br /></i>
<i>A pre-built binary as a 7zip file is located here on my personal website:</i><br />
<i><br /></i>
<a href="http://www.duatiu.com/rd.7z"><i>renderdebug 7-zip binary.</i></a><br />
<i><br /></i>
<br />
<ul>
<li><i>You must have a Windows machine with a DirectX 11 compatible video card to run the tool.</i></li>
<li><i>Run the batch file 'show_bitcoin.bat' in the root directory.</i></li>
<li><i>The application uses Maya style navigation controls. You must hold down the Alt key first before the mouse will rotate and control the camera. Use the middle-mouse button to pan.</i></li>
<li><i>You can go the tab marked 'World' and turn off the checkbox to show the background grid which is on by default.</i></li>
<li><i>On the tab marked 'Debug Recording Playback' you have VCR style controls that will let you scrub through the data or single frame advance and rewind.</i></li>
<li><i>If you have problem running the tool, maybe because I forget some dependent DLL or something, let me know. mailto:jratcliffscarab@gmail.com</i></li>
<li><i>Here is a link to a YouTube video showing the distribution of bitcoin value on the blockchain throughout it's entire history. <a href="http://youtu.be/SbA913dLfYU" style="background-color: white; border: 0px; color: #1b7fcc; cursor: pointer; font-family: arial, sans-serif; font-size: 11px; line-height: 14.300000190734863px; margin: 0px; padding: 0px; text-decoration: none;" target="_blank">http://youtu.be/SbA913dLfYU</a></i></li>
<li><i>Here is a link to a YouTube video showing the distribution of non-zero balance bitcoin public key addresses over time. <a href="http://youtu.be/e074zH4vfWE" style="background-color: white; border: 0px; color: #1b7fcc; cursor: pointer; font-family: arial, sans-serif; font-size: 11px; line-height: 14.300000190734863px; margin: 0px; padding: 0px; text-decoration: none;" target="_blank">http://youtu.be/e074zH4vfWE</a></i></li>
<li><i>If you enjoy using the tool and appreciate the effort to produce all of this data, please be kind enough to send me a bitcoin tip. It's not so much that I need the money but rather that it's good feedback so I know this work is appreciated by the community.</i></li>
<li><i><a href="https://blockchain.info/address/1BT66EoaGySkbY9J6MugvQRhMMXDwPxPya">Bitcoin Tip Address</a> </i><a href="https://blockchain.info/address/1BT66EoaGySkbY9J6MugvQRhMMXDwPxPya" style="background-color: white; color: #0088cc; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 13px; line-height: 20px; text-decoration: none;">1BT66EoaGySkbY9J6MugvQRhMMXDwPxPya</a></li>
</ul>
<br />
<br />
Here are some summary statistics current of as May 16, 2014. Please be cautious about trying to infer a great deal from associating public key addresses with individual people. So many of these belong to large exchanges and companies who are, in turn, managing many people's funds off-blockchain.<br />
<br />
<br />
* Total Blocks: 301,044<br />
* Total Transactions: 38,881,761<br />
* Total Inputs: 89,839,209<br />
* Total Outputs: 100,534,409<br />
<br />
* Found 35,989,045 addresses which have ever been used.<br />
* Found 33,074,177 addresses with a zero balance.<br />
* Found 1,386,411 'dust' addresses (less than 1mbtc) with a total * balance of 171.26523 BTC<br />
* Found 1,217,994 addresses with a balance greater than 1mbtc but less than 1btc, total balance 113,806 btc<br />
* Found 199,264 addresses with a balance greater than 1btc but less than 10btc, total btc: 504,628<br />
* Found 96,553 addresses with a balance greater than 10btc but less than 100btc, total: 3,461,562<br />
* Found 13,128 addresses with a balance greater than 100btc but less than 1,000btc, total: 2,992,605<br />
* Found 1,419 addresses with a balance greater than 1,000btc but less than 10,000btc, total: 3,157,431<br />
* Found 98 addresses with a balance greater than 10,000btc but less than 100,000btc, total: 2,290,441<br />
* Found 1 addresses with a balance greater than 100,000btc, total: 144,341<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeP9k-pdnCQu332kHucUGJUrqd-Ey-yK96dmZYBzhu4avGA_otye7HhZF1xOJRmOrYsnVcrn5aWd4tyhObjJgtxYNPPJdo9sFq_j8-tGOCHsBWAsXj1q7UGbf40U1WdUE8foM8vw/s1600/bitcoin-age.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeP9k-pdnCQu332kHucUGJUrqd-Ey-yK96dmZYBzhu4avGA_otye7HhZF1xOJRmOrYsnVcrn5aWd4tyhObjJgtxYNPPJdo9sFq_j8-tGOCHsBWAsXj1q7UGbf40U1WdUE8foM8vw/s1600/bitcoin-age.png" /></a></div>
<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">* Bitcoin value distribution based on age.</span><br />
<span style="font-family: Courier New, Courier, monospace;">* Age Value Key Count</span><br />
<span style="font-family: Courier New, Courier, monospace;">-----------------------------------------------------</span><br />
<span style="font-family: Courier New, Courier, monospace;">* One Day : 248,585 BTC : 6,732</span><br />
<span style="font-family: Courier New, Courier, monospace;">* One Week : 280,393 BTC : 9,355</span><br />
<span style="font-family: Courier New, Courier, monospace;">* One Month : 677,688 BTC : 21,200</span><br />
<span style="font-family: Courier New, Courier, monospace;">* 1-3 Months : 2,086,637 BTC : 49,382</span><br />
<span style="font-family: Courier New, Courier, monospace;">* 3-6 Months : 2,779,152 BTC : 79,066</span><br />
<span style="font-family: Courier New, Courier, monospace;">* Six Months to One Year : 1,755,184 BTC : 36,037</span><br />
<span style="font-family: Courier New, Courier, monospace;">* One to Two Years : 1,721,166 BTC : 40,848</span><br />
<span style="font-family: Courier New, Courier, monospace;">* Two to Three Years : 850,632 BTC : 28,523</span><br />
<span style="font-family: Courier New, Courier, monospace;">* Three to Four Years : 895,958 BTC : 11,518</span><br />
<span style="font-family: Courier New, Courier, monospace;">* Over Four Years : 1,449,591 BTC : 27,831</span><br />
<div>
<br /></div>
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzuKsinCGmvqATh6E5PkaxArsLCU46pLVC_VyKH9URk9QZtxfZZI7RRweeWg2ERN6BiBV-UKzUx52VVTIMzvJngKS7mKb_Wgi0-wQVati4do4OQtPmBWmWCBzOjyeFa5PsvmXdyQ/s1600/blockchain-parser-qr-code.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzuKsinCGmvqATh6E5PkaxArsLCU46pLVC_VyKH9URk9QZtxfZZI7RRweeWg2ERN6BiBV-UKzUx52VVTIMzvJngKS7mKb_Wgi0-wQVati4do4OQtPmBWmWCBzOjyeFa5PsvmXdyQ/s1600/blockchain-parser-qr-code.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">TipJar: <a href="https://blockchain.info/address/1BT66EoaGySkbY9J6MugvQRhMMXDwPxPya">1BT66EoaGySkbY9J6MugvQRhMMXDwPxPya</a></td></tr>
</tbody></table>
--------------------------------------------------------------------------------------------<br />
<br />
<h2 style="text-align: center;">
Introduction to the RenderDebug Debug Visualization Library</h2>
<br />
Have you ever been working on a command line or server application and desperately needed to display some debug visualization data but couldn't because you had no access to a graphics rendering library? Or, how about you were working in a large complex graphics application, like say a full game engine, but you just needed to be able to visualize something that might be happening in the physics engine and displaying that data in the main window was incredibly difficult to do?<br />
<br />
RenderDebug is a library that I have kept around for a very long time. I can hardly remember when I wrote the initial version of it. I have released it as part of many of my open-source projects in the past, in various forms, but I don't think I have ever written it up before.<br />
<br />
RenderDebug is a debug visualization API. It is not about doing high-end rendering, or anything complex. It's entire purpose in life is to display debug information like 3d text, boxes, spheres, rays, arrows, lines, triangles, and the like. It's a pretty clean and simple to use API. It uses 'const float' for all parameters passed in, so it does not enforce or require any particular math library.<br />
<br />
Here is a<a href="https://code.google.com/p/renderdebug/source/browse/trunk/public/RenderDebug/RenderDebug.h"> link to the single header file </a>needed to use the RenderDebug system. I will try to add some better documentation soon, it's a bit sparse for the moment.<br />
<br />
The API is contained in a single header file with no other dependencies. Currently I am only providing an implementation as a plugin DLL for windows. It actually does compile for nearly every platform known to man, but I cannot provide full source at this time due to it having some dependencies on a shared multi-platform source library that I cannot release at this time. Perhaps, in the future, once the license status of the platform abstraction library becomes more clear, I will release the full source as well. However, at least on Windows, you can use the plugin fine without needing direct access to the source code. <br />
<br />
For now what I am providing is the library as a plugin style DLL that can be demand loaded into any Windows application, console or otherwise, for both 32 and 64 bit. I may release a Linux shared-object file as well at some point in the future.<br />
<br />
So, here are some of the things which make this debug visualization library unique and interesting.<br />
<br />
<h2 style="text-align: center;">
RenderDebug Features</h2>
<ul>
<li>It provides a rich set of debug visualization primitives like lines, triangles, boxes, spheres, capsules, rays, arcs, frustum and matrix visualization.</li>
<li>It uses a 'const float' style API so it enforces no specific math library, though the layout of matrices, quaternions, and vectors passed in as const float should conform to a fairly standardized layout (OGL/D3D).</li>
<li>It can render things in wireframe, solid shaded, or both solid-shaded with a wireframe outline</li>
<li>It provides support for oriented 3d text</li>
<li>If your application provides a valid view and projection matrix, it can also do screen-space 3d hud style rendering and it can orient text and quads to the camera.</li>
<li>It uses a stack-based state system for simplicity and efficiency. For example, rather than having to pass the color into every single API call, instead you just set the current color and all draw commands after that are assumed to use this state. The state is maintained in a state-stack, so you can push the current state, change the color, display time, or other global state settings, do some draw commands, and then pop the state restoring everything back to it's original values.</li>
<li>It understands the concept of time! Many debug visualization libraries might offer you an API to, say, draw a line, however, they will usually only draw that line for one frame. With RenderDebug any debug visualization primitive has a 'life time'. Sure, you can draw a line for just one frame, but you can also draw a line and indicate it should stay on screen for 60 seconds. Why is this important? Because often when doing debug visualization you are trying to see a transient event. Maybe you want to visualize a raycast hit location event that happens discretely. If you only draw these kinds of contact events for one frame you will usually miss seeing them.</li>
<li>The RenderDebug library does not do any actual rendering itself. Instead it buffers up all of the debug visualization primitives to be streamed in a variety of different ways. Debug visualization can be streamed to a file, over a communications channel to another application, or it can provide raw geometry that you can submit to your own graphics engine.</li>
<li>The RenderDebug library also supports debug logging messages as well as remote commands if you are using it in client/server mode.</li>
<li>It also supports draw groups. This allows you to create a set of debug commands which get cached by id, so that you don't have to update them every single frame. Think of them like macros for draw commands. Using the setPose command you can render a draw group, multiple times, with different transforms.</li>
</ul>
<h2 style="text-align: center;">
Example Use Cases</h2>
<div>
<br /></div>
<div>
Here are some example use cases where it makes a lot of sense to use the RenderDebug library</div>
<div>
<br /></div>
<div>
<ul>
<li><b>Your Own Graphics Application</b>: Probably the most obvious use case, is to simply take advantage of it for general debug visualization in your own graphics application. If you have any graphics application of your own, but have not yet implemented a rich debug visualization library, then why not use RenderDebug? All you have to do is implement a single callback interface called 'RenderDebugInterface' and you can embed the entire rich debug visualization library directly into your own application. The RenderDebugInterface simply requires that you implement four methods. One to render lines, one to render triangles, one to log a debug message and, finally, one to process a command from a remote client.</li>
<li><b>A Server Application : </b>One use case that is really powerful is for servers. Let's say you are working on a massively mutiplayer online game and you are implementing the server. While the server has to represent AI and players and world geometry, rarely do servers have any graphics front end. Servers are typically console applications with no way, other than some debug log messages, to have any idea what is happening inside. With the RenderDebug library you can use the front-end DebugView application to 'talk' to your server in real-time. You can send it commands and the server can make calls into the RenderDebug library to provide real-time debug visualization of it's internal state. I did exactly this when I was working on an MMO server years ago and it proved to be an incredibly powerful tool.</li>
<li><b>A Mesh Processing Tool : </b>Let's say you are working on a mesh processing application, for example the convex decomposition tool that I have previously published. Rather than having to spend days creating a graphics application which can test the mesh processing tool and display the output, instead you can just easily add the RenderDebug library to your console mesh processing application and get a debug visualization of the output. You can even create multiple-frames of debug visualization data. For example with the HACD console application, on the first frame I debug visualize the original triangle mesh. On the second frame I debug visualize the convex decomposition, color coding each convex hull. Then, on the next frames I show each hull getting added to the result. On the final frames I create an 'exploded' view of the convex decomposition. You can see an example in this YouTube video link. <a href="https://www.youtube.com/watch?v=U6MRe36YKIk&feature=youtu.be">https://www.youtube.com/watch?v=U6MRe36YKIk&feature=youtu.be</a></li>
<li><b>A Simulation Tool : </b>If you are working on a physics or other kind of simulation project, for example it could be doing AI processing, you can perform all kinds of debug visualization of the internal state of your simulation without needing to drag in any graphics or rendering engines. This can make prototyping go much, much, faster.</li>
<li><b>General Statistical Analysis</b> : Let's say you have a bunch of data that you are trying to analyze. Sure, you can export it as a CSV and import it into a spreadsheet application so that you can generate some graphs. In fact, that is what you should do. However, what if you want to see how a particular data set evolves over time? Or let's say that your data contains a great deal of visual information, like the kind that cannot easily be conveyed in a conventional graph? With the RenderDebug library you can implement your own way to visualize a data set, including evolving it over time. I did exactly this the other day. For those who follow this blog, they know by now that I have spent a lot of time doing analysis on the bitcoin blockchain. A lot of what interests me these days is forensic analysis on how the bitcoin blockchain has evolved over time. So, in part to test the RenderDebug library, I quickly revised my bitcoin blockchain parser tool to create a debug visualization bar graph of the distribution of bitcoin value by age and then created one frame for every single day in the history of bitcoin. The results were very cool and would be even cooler if I spent more time creating a better looking graph. Here is a link to the YouTube video which shows how the bitcoin blockchain value distribution has evolved over it's entire history. Here is the link. <a href="http://youtu.be/SbA913dLfYU" style="background-color: white; border: 0px; color: #1b7fcc; cursor: pointer; font-family: arial, sans-serif; font-size: 11px; line-height: 14.300000190734863px; margin: 0px; padding: 0px; text-decoration: none;" target="_blank">http://youtu.be/SbA913dLfYU</a></li>
</ul>
<div>
<br /></div>
<h2 style="text-align: center;">
Future Work</h2>
<div>
<br /></div>
<div>
One of the key features of the RenderDebug library is not yet implemented, which is the ability to do client/server communications in real-time. I am still working on that. I expect it to be done in the next week or so. It will use inter-process communication via a shared block of memory. In this mode you will be able to run your own project, say a game engine, and in real-time interact with the standalone DebugView visualization application. You will be able to not only send log messages and debug visualization data from your game engine to the viewer application, but you will also be able to send console commands between the two applications as well. In fact, you can actually use DebugView as your primary mechanism for debug visualization and console commands for your project if you want! For example, you could send a command from DebugView to your game engine to enable some particular debug visualization feature and essentially control the game engine remotely. In fact, I even plan to send mouse picking and drag events over the API too. This should allow me to develop fully interactive PhysX simulation demos using DebugView as a front-end tool to interact with the simulation.<br />
<br />
For readers familiar with <a href="https://developer.nvidia.com/physx-visual-debugger">PVD</a>, the <a href="https://developer.nvidia.com/physx-visual-debugger">PhysX Visual Debugger</a>, you may see a lot of similarities here. Both DebugView and <a href="https://developer.nvidia.com/physx-visual-debugger">PVD </a>tackle very similar problems. However, while <a href="https://developer.nvidia.com/physx-visual-debugger">PVD</a> is focused exclusively to the task of visualizing what is happening inside the <a href="https://developer.nvidia.com/physx-sdk">PhysX SDK</a>, DegugView is a general purpose debug visualization viewer. It deals only in debug visualization draw commands, log messages, and command streams.<br />
<br />
There are also some parallels here with the kinds of features offered by projects like <a href="http://www.radgametools.com/telemetry.htm">Telemetry</a> from <a href="http://www.radgametools.com/">Rad GameTools</a>. Now, don't get me wrong, I am a huge, huge, fan of <a href="http://www.radgametools.com/telemetry.htm">Telemetry</a> and I highly recommend it to anyone who is concerned about performance analysis of their product. However, I would like to see a remote debug visualization feature like RenderDebug in <a href="http://www.radgametools.com/telemetry.htm">Telemetry</a> too, and I've talked to Brian about this in the past but it's never been a super high priority before.</div>
</div>
<div>
<br /></div>
<div>
I realize that it would be both useful and valuable for this system to also support inter-process communications via TCP/IP so that you could get remote debug visualization from console games or remote servers. Unfortunately, that feature won't be coming right away even though I have implemented it before in previous versions of the tool. My immediate short term need it to get remote-debug visualization from game engines I am running from the same machine I am on.<br />
<br />
<div>
Future work on this project may include the following features, though there is a strong possibility I will not do all of them unless I have a specific reason to do so. </div>
<div>
<br /></div>
<div>
<ul>
<li>Add support for real-time interactive visualization on the same machine using a client/server relationship and inter-process communications via shared memory.</li>
<li>Add support for dynamic interaction, ray-picking, for example, so that it the DegugView application can be used to control a real-time ragdoll physics simulation.</li>
<li>Implement compression for the channel of debug visualization data to lower the bandwidth requirements</li>
<li>It might be interesting to make a way for scripting languages to talk to the RenderDebug API so that people who work in Perl or Python or any one of those other whizzy scripting languages could get remote debug visualization out of them. I don't use any of these scripting language myself, so I probably will not do this any time soon, though I can see some valuable uses connecting it to an embedded scripting language in a game engine.</li>
<li>Release the RenderDebug source code open-source and get a build that works using Shared-Object files on Linux</li>
<li>Release an simple open-source version of a DebugView application to demonstrate how to bind to a graphics layer fro rendering.</li>
<li>Implement a TCP/IP mechanism so that the DebugView application can connect to remote applications on other machines like a game console for example.</li>
</ul>
</div>
</div>
<div>
<br />
<h2 style="text-align: center;">
The DebugView Application</h2>
<br /></div>
<div>
The DebugView application is what you use to play-back debug visualization recording files. Just launch it and pass as a command line argument the name of the recording file. Or you can use the 'File' menu to browse for a recorded debug visualization file. The DebugView application uses Maya style controls to navigate, so use the 'Alt' key in conjunction with the mouse to move the camera around.<br />
<br />
It requires a Windows machine with a DirectX 11 capable graphics card. This is not because it needs some high-end graphics card for any particular reason, it's just that the render library I built the tool against is DX11 and I figure most anyone these days has a DX11 card by now anyway. <br />
<br />
I have tested this application on a number of machines but there is still every chance it might fail to run on some machine configurations. If that happens, let me know and I will see if I can figure out what is going on.<br />
<br />
<br />
<h2 style="text-align: center;">
Related Google Code Projects</h2>
<br />
Here are the projects that are relevant to this article:<br />
<br />
<br />
<ul>
<li><a href="https://code.google.com/p/renderdebug/"><b>RenderDebug</b></a>: The main renderdebug Google Code project page. Contains the DLL plugin and two sample applications, ObjView which reads in a wavefront OBJ file and TestRenderDebug, which created an animated debug visualization. Here is a short YouTube video clip showing the visualization produced by TestRenderDebug: <a href="http://youtu.be/B__h--QGUXM" style="background-color: white; border: 0px; color: #1b7fcc; cursor: pointer; font-family: arial, sans-serif; font-size: 11px; line-height: 14.300000190734863px; margin: 0px; padding: 0px; text-decoration: none;" target="_blank">http://youtu.be/B__h--QGUXM</a></li>
<li><a href="https://code.google.com/p/hacd/"><b>HACD</b></a>: The hierarchical approximate convex decomposition app which now can build for Linux and also outputs a debug-render visualization of the results on Windows. Here is a YouTube video showing an example. <a href="https://www.youtube.com/watch?v=U6MRe36YKIk">https://www.youtube.com/watch?v=U6MRe36YKIk</a></li>
<li><a href="https://code.google.com/p/blockchain/"><b>blockchain</b></a> : And finally the bitcoin blockchain parser which now has an option on the Windows version to generate a debug visualization output of the blockchain over time. Here is a link to a YouTube video showing the output. <a href="http://youtu.be/SbA913dLfYU" style="background-color: white; border: 0px; color: #1b7fcc; cursor: pointer; font-family: arial, sans-serif; font-size: 11px; line-height: 14.300000190734863px; margin: 0px; padding: 0px; text-decoration: none;" target="_blank">http://youtu.be/SbA913dLfYU</a></li>
</ul>
<div>
<br /></div>
<div>
<ul>
</ul>
</div>
<br />
<br />
<br /></div>
John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com0tag:blogger.com,1999:blog-23430315.post-85639068399884443992014-04-09T12:52:00.000-07:002014-04-14T15:54:16.292-07:00How to create a hardened API for middleware<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIhWlaNl2XJdCkqF-HH7dd0XLOo_K0EvD82dMl3-jVMnNz0zGbjpNpmOgwsVF65qQDq4xSZiThP5eGVgOkwkNRF4AlXxUhV1IL400hC7VwDUvHwLE7o3MHFsfihm2U9bXoeRJtFw/s1600/hardened-heart.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIhWlaNl2XJdCkqF-HH7dd0XLOo_K0EvD82dMl3-jVMnNz0zGbjpNpmOgwsVF65qQDq4xSZiThP5eGVgOkwkNRF4AlXxUhV1IL400hC7VwDUvHwLE7o3MHFsfihm2U9bXoeRJtFw/s1600/hardened-heart.jpg" height="400" width="640" /></a></div>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Being a middleware developer is challenging enough to begin with, especially since it is such a tough business model, but support can make your life a living hell. The advantages for the customer are clear, developing software is expensive and risky and licensing middleware reduces risk by adopting a proven piece of technology. For the licensee, their software engineers will have to spend less time reinventing the wheel, doing research and development and can instead focus on the specific product they are trying to create. </span></div>
<b id="docs-internal-guid-ee6047ae-47ef-752b-cfda-6e9f833d3383" style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Today’s modern game engines make heavy use of licensed middleware. Gone are the days of the ‘not-invented-here’ syndrome. Sure, there are a few holdouts who want to do everything their own way, but they are few and far between. Why would you write your own video codec, compression engine, or sound system? What would be the point? These technologies are practically commodities at this point.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Some pieces of middleware are a bit more controversial. In particular are libraries designed to solve physics and graphics problems. Rarely do these ever just drop easily into an integration. There are often unrealistic expectations on the part of the customer. Frequently they don’t realize that even though they have licensed this middleware they still must account for the time it takes to integrate it, as well as learn how to use it to implement specific features their product requires. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The biggest problem middleware providers face is support. This goes beyond simply trying to explain to a developer how to use the API, it is the fact that now that your software is deeply embedded in someone else's product, and if their product crashes anywhere inside of your code, then, Houston, you have a problem. In fact you may have a very, very, very, big problem. Even if the developer passed complete garbage data into your middleware, even if they abused your middleware worse than Bobby Brown, if it crashes anywhere in your code it almost always instantly becomes your fault. </span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Once your middleware has become part of someone else's product, you are for practical purposes usually an unpaid member of their development team. You may often find yourself in the situation of being used as a scapegoat for why things are going wrong. If a developer can't get something to work the way their manager wants it to, or there are other problems, they can just blame the middleware since those engineers aren't around to defend themselves. Frankly, these kinds of political issues arise frequently. </span></div>
<b style="font-weight: normal;"><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Something else to be aware of is that, frequently when people license a piece of middleware they may not understand the underlying technology very well themselves. Often junior level programmers may be trying to use your API in ways that it was never intended to be used or you never could have imagined. They may call your API from the most unexpected places and in the most unexpected ways. Once your middleware has been licensed to several customers you may feel like you never get to write a new line of code ever again. You may feel as though you are under constant attack and bombardment from your customers which, in actuality, you probably are. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Here are some of the issues you will deal with in support.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Their product will crash in your code, which makes it immediately ‘your fault’</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They will invoke your API from a myriad of different threads, even if your documentation says they shouldn't do that.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They will pass bad pointers into routines and blame you when the code crashes.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They will pass in bad floating point numbers which will then, in turn, produce infinities, and expose bizarre behavior and crashes.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They will pass in what are technically considered ‘valid’ floating point numbers but they will be wildly out of range, either massive in size or incredibly tiny; creating all kinds of pathological behavior.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They may create hundreds of thousands of instances of objects in a system designed to handle no more than a few hundred.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They may set constants and other tuning parameters to outrageous values that cause the system to consume insane amounts of CPU and then blame you because your middleware is so slow.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They may create thousands of objects but never release them.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They may create thousands of objects all at the origin since, at object creation time, they don’t know their position yet.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They will pass in normals that are not normal.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They will pass in matrices which are invalid.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They will pass in quaternion rotations which are completely invalid.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They will pass in triangle mesh geometry with degenerate faces, and all manner of corruption.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They will pass in convex hulls which are completely inappropriate for real-world use.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They will report impossible to reproduce problems.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They will report problems which only occur, at random, after their product has been running for 10 hours straight.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">They will report problems but give you no way to debug or support them short of getting on an airplane and flying physically to their location.</span></div>
</li>
</ul>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I could go on and on, but you get the idea. Like I said, the business model for middleware is challenging enough, but the support nightmare makes it a largely thankless task. What is a middleware provider to do?</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Treat your customer as your enemy. Imagine your customer is actively trying to write malware to attack your API. While they may not consciously be doing that, in the practical case that is essentially what is happening.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There is a solution to this problem, but it is rather extreme. You do not, necessarily, have to go the full extreme route to address these support issues. You can do something in between and live with the risks that you leave yourself open to. The rest of this article is going to discuss the extreme view of implementing a fully hardened API and what that requires.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I also do not talk a great deal about unit-testing here. Obviously you should have a robust set of tests and samples for your SDK, that is a given. However, unless your API is very simplistic, it is doubtful that you will even begin to scratch the surface of how end-users will use your system. Yes, write as many tests as you can but also realize the limitations of this testing coverage.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">One thing I strongly recommend is that if your middleware is designed to be used by large scale development projects, then do not release it until it has been thoroughly integrated and tested in at least one large scale development project. Sandbox and sample testing is so unrealistic as to be relatively useless.</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In my career as a software engineer I have seen numerous products released that were designed to help developers with some set of problems that, as soon as you applied it to a real-world massive project, rolled over and died immediately.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To be clear, I am not saying that you </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">should</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> do all of these things. What I am saying is that if you wanted to make a perfectly hardened API then you would have to do so; especially since anything less does expose you to some element of risk. The actual solution is probably to adopt some of these recommendations, but not all of them everywhere at all times.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The following rules apply </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">ONLY </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">to your public API. Internally, you can code however you wish and to any standard that you wish. These guidelines only refer to how your end-user, your customer, is allowed to interact with your middleware. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Rule #1</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> : </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Never use pointers, EVER! </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> If your API ever accepts a pointer, anywhere, then you are implying a level of trust with the developer that is not warranted. You are trusting that the developer is only ever going to pass you a valid pointer. You might even think, hey, this is a pointer to an object that I allocated so it must be safe, but that still doesn't cut it. You would now be trusting the application to pass you back, unmolested, the same pointer you created and you don’t know that this is true. The simple fact is that you cannot allow the end user to pass you a pointer that your code then operates on. If you do, and they pass you a bad pointer, then your code will crash when you dereference it and, when your code crashes, it’s not their fault, it’s now your fault. This is more than just testing for null pointers which, of course, you should do. This is about any potentially random value a pointer could be assigned to. The pointer could be bad because it was in an uninitialized data section, it could be bad because it refers to an object which has already been deleted, or it could be bad simply because the application developer trashed it. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The same thing goes for references. References are pointers too, so don’t think that somehow lets you off the hook here.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You might be wondering now, if you cannot ever use pointers, what do you use instead? Instead of pointers you should use ‘handles’. A handle is simply an integer value which refers indirectly to a pointer. In any use case where your API might normally return a pointer, instead you should return a handle (integer) instead.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">When the user passes a handle back to you (instead of a pointer) you can completely and safely validate this handle with zero risk. Your own SDK will maintain a handle table, that maps an integer to a pointer. First you make sure that the integer passed in is within range and represents a valid handle/pointer mapping. If it does not, then you simply do not attempt to operate on it, and return an error code and/or issue an error message to the developer via whatever logging mechanism your SDK implements. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Even if the handle passed in is valid, it may not necessarily be pointing to the correct type of object, so your code should validate this as well before dereferencing it.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Rule #2 : Never use C++ non-static methods EVER!</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> This may not seem obvious at first, but this is actually just an extension of Rule #1 to never use pointers. It is pretty much impossible to use C++ without using pointers, unless all of your methods are static which wouldn't really make much practical sense. Whenever you call a method on a C++ class there is an implied pointer known as the ‘this’ pointer. If any part of your API uses C++ methods it violates the first rule to not use pointers, for all of the same reasons cited above.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You can use C++ classes or flat structs so long as none of the member variables contain pointers or non-static methods. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Rather than further confusing things, the best rule is to make your entire API be straight vanilla C. These functions (not methods) will be physically linked into your application. The customer will get a link error if they use the API incorrectly, but there is no risk at runtime.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It is important to remember, once again, that this is only in reference to your public customer facing API. Your actual implementation code can use all of the C++ and object oriented goodness that you want, you just can’t expose any of that to the end user.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Rule #3 : Never use the STL, Boost, or other dependencies in your public API. </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">This is basically the same as rule #2. These APIs are all based on C++ and imply way too much trust about the format and validity of the data being passed. Once again, you can use them in your own internal implementation code if you wish, so long as you are willing and ready to accept any risk that they pose.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Rule #4 : Never trust a floating point number passed into your API </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">: You would be amazed at the world of trouble that this can cause. Floating point numbers are almost worse than garbage pointers being passed into an API because they are more insidious in some ways. If you dereference a bad pointer, you will generally get a crash right then and there. However, if you reference a garbage floating point number, instead of getting a crash, you will likely just get a garbage result. One bad floating point number used in a calculation will then infect the result and get passed on down the line. (Infect is actually a good way to put it, bad floating point values are like viruses, they infect the rest of your systems making each one sick and spreading it to others.)</span><br />
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Soon your entire system is riddled with garbage floating point numbers throwing every calculation off. None of these may even necessarily cause immediate crashes but, instead, you get pathological behavior and things may simply stop working or behave in some very, very, bizarre ways. One example of how this sort of thing might cause an infinite loop is as follows. Let’s say that you have some sort of a hierarchical bounding volume tree that you traverse. Now, let’s say you are traversing it but with a bad floating point number. This could break the tree traversal in all sorts of unexpected ways since floating point compares may always fail when the expected behavior is that one side is always greater while the either side is always lesser. This could lead to infinite loops or missing results.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Another chance for bad floating point numbers to crash your code is if you convert a floating point number back into an integer which you then use to index an array with a bogus result.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">By default most applications do not generate an exception when a bad floating point number is accessed. Exceptions can be enabled, but this can have unintended side effects. In general it is not valid for your middleware to change a global system wide setting like the floating point control word. You might enable exceptions to catch problems in your own code but then the application you are running inside of might suddenly start crashing or, worse yet, just turn them back off again. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Another thing to be aware of is how the floating point unit handles numeric precision and rounding behavior and the fact that it is not always preserved. It can be changed under your feet without you knowing about it! For this reason your API also needs to confirm that the floating point unit is always in the precision and rounding mode that your SDK expects and if it is ever changed, you should issue an error message and stop operating on the input data. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Since your code should never change the floating point unit state and, more to the point, our goal here is to never, ever, crash, then enabling exceptions that we do not handle gracefully is not really the solution. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Instead, the recommended course of action is to verify that each and every single floating point number passed into your API is valid by using the ‘isfinite’ check in the standard library.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In addition to making sure that all floating point numbers are valid, you should also make sure that they are within a ‘reasonable’ range. Just what a reasonable range means is specific to the context of your problem space. Let’s say this is for some kind of a game which has a world size that is never larger than 8 kilometers and is represented in meters. In this example any floating point number representing a ‘position’ that is outside of that range would be considered invalid. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Another thing to check for is normalization. Let’s say one of the values being passed into your API is supposed to represent a vector normal. All three floating point numbers could be perfectly valid, but that doesn't mean it is a valid normal. The same thing goes for matrices and quaternions.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">To deal with these problems it requires that you perform full floating point validation not just for invalid numerics, but also numerics which are within range and appropriate for the use case.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">By now you are probably wondering, isn't all of this going to be incredibly expensive and degrade performance?</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The answer to that question is, probably yes. Here are ways to deal with this.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Your SDK can have multiple build configurations. One which is hardened to an extreme degree, a ‘checked’ build as it were, and one which does no, or fewer, checks and is typically called a ‘release’ or ‘shipping build. </span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Your SDK can have different levels of validation. You can even implement this as a run-time option; essentially by having each API call redirect to a different implementation that does the degree of validation required. This solution is a bit more difficult to implement but probably worth it, as it allows the developer to swap out which degree of checking they perform simply by changing a .INI file without requiring a different build of the game.</span></div>
</li>
</ul>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Rule #5 : Never trust any other parameter passed into your API</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> : While we have discussed the validation of floating point numbers, the same degree of validation is required for all other parameters, whether it is an integer, or enumeration, or the contents of a data structure. All parameters passed into any method should be completely validated as much as is necessary to make sure they are in a reasonable range and express expected values.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Rule #6 : Always be thread safe! </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In this day and age you can never assume that any API call is not going to be interrupted by a call to another function, or even the same routine you are already executing, but from another thread. All API calls should have, at the minimum, a mutex to make sure that you don’t get burned by reentrancy and may, in fact, need a thread safe command queue if this is a common usage pattern.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Exceptions to the Rule</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The rules above are required to create a nearly perfectly hardened API which can never, ever, be crashed because the user submitted bad input data. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It is also possible that even though the API is hardened, the application could still just stomp directly on some of your internal state data. There are some things you can do to detect and protect against this sort of thing but it is generally considered impractical. Short of making your SDK act as a kernel mode driver, some of these crazy abuses cannot be easily guarded against.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">It is possible that they could still abuse your API by making unreasonable demands and requests on it, causing it to run very slowly or otherwise consume a great deal of CPU. </span><span style="font-family: Arial; font-size: 15px; line-height: 1.15; white-space: pre-wrap;">Your code could have checks to detect this situation and allow it to bail out if a particular routine takes too long to execute and treat that as a warning or error.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">At the end of the day, no matter how much you want to harden your API, these rules may ultimately be impractical. If the user needs to add millions of data items and the only way to do so is by making a million discrete API calls which are fully validated, this simply may not work. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You can break any of the rules above, just so long as you are aware of and willing to accept the risk that comes with it.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">You could take the approach of having the vast majority of your API be fully hardened yet expose just a handful or routines which require high-speed data transmission to occur. This would make things mostly safe most of the time and the non-hardened API could be thoroughly documented in detail and would hopefully be a risk you could manage.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Features to make your life easier</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There are a couple of features your SDK probably needs to keep your life from becoming a living hell. As soon as a customer has a problem that only happens in their product and cannot be reproduced outside of it, you will find yourself in a difficult situation. Usually you will not have source code access to their product and, even if you did, learning how to sync, build, and run their product could consume days, if not weeks, of your time.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">There are a few key features you need to implement in your SDK to help deal with this. Most of these features will be much easier to write so long as you have used the API hardening rules above.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Logging :</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Your SDK should support extensive logging features. This goes beyond simply logging warnings, errors, and other poor usage patterns but even go so far as to log the contents of every single API call. You can, and should, support multiple levels of data logging detail. The data-logging should be easily human readable and act as a useful aid in diagnosing runtime problems. A lot of times just a quick look at a log can reveal to the support engineer, holy shit, they are calling the API this crazy way!! </span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">API record / playback :</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> This is probably the single most important piece of technology you should provide. No matter how long it takes you to implement a robust record and playback system, it will save you far more time than you would otherwise spend on product support. The concept is simple. Every single time the application makes a call into your SDK, you write the entire contents of that API to a stream (flushing the stream immediately). This process has to be so fast that it does not interfere with the operation of the actual product. </span><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;">Once you have a recording you should also have a separate application of your own which can play back that data. Imagine that your customer has a difficult to reproduce crash that only happens semi-randomly after running their product for an hour. Your hopes of debugging this in the traditional way would be bleak, but with a sufficiently powerful record and playback system there is real hope. The QA person would run the product with the recording feature enabled up until the problem occurs. It could be a crash (which really should not happen if your API is sufficiently hardened) or it could be a performance or behavior problem. They should be able to annotate the recording with metadata tags as well (text tags which indicate what was happening in the application over time). Once they have captured this file they can simply send it to you. With that file, and without needing access to their source code or product, you can reproduce this issue </span><span style="font-family: Arial; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">exactly </span><span style="font-family: Arial; vertical-align: baseline; white-space: pre-wrap;">and reproduce it deterministically every single time over and over again. No matter how much time you may spend developing this API capture and playback feature all of that effort will be paid back the first couple of times you can fix an issue using it. Remember that the API record/playback system must be able to take multi-threading into account. Many problems which happen in modern day software results from issues related to multithreading and reentrancy. For really tricky multi-threading issues there is a chance that a playback will not be able to reproduce the problem, so you should be aware there are some limitations with this approach.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Debug Visualization : </span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Depending on the type of SDK you are creating it may be valuable to have a debug visualization tool which can render a meaningful picture of what is happening ‘inside’ of your engine.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Performance Tracking :</span><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> Often times the reports of problems you will get from customers is not that your SDK is crashing but, instead, that it is consuming too many resources; be that CPU, GPU, or memory for example. Embedding robust and detailed performance tracking into your product can be invaluable. In this case, rather than writing your own from scratch, I strongly recommend you consider yet another piece of middleware, which is </span><a href="http://www.radgametools.com/telemetry.htm" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">Telemetry from Rad Game Tools</span></a><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">. It is an incredible tool which will allow you to easily embed many of these logging, analysis, and visualization features.</span></div>
</li>
</ul>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b><br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: bold; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Conclusion</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In conclusion I am not saying that you have to do these things, or even that you should. The point of this article is simply to present the extent and degree to which an API can be hardened against attack, either intentionally or not, by end users. </span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">As extreme as all of these things sound, it is important to remember the real-world problems they are designed to address. If you develop a non-hardened API to be used by a general population of not necessarily expert programmers, you are going to run into a lot of problems. If you would like to spend more of your time developing cool technology and less of it in hopeless support tasks, tasks which most of the time are not even issues with your software, then you may want to consider some of these strategies.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If a lot of this may sound hypothetical but I assure you that it is not. I have developed a number of APIs in my career targeted at various audiences. Here are the real problems I have encountered, quite frequently in fact.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have frequently been passed garbage pointers into routines and told it was my fault when it crashed in my code.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have frequently been passed bad floating point numbers.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have frequently been passed unreasonable floating point numbers that were far out of practical range.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have frequently been passed invalid parameters, out of range and otherwise bogus.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have frequently had an API get blindly called from all kinds of different threads in ways I never imagined anyone would, or could, do so.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have frequently had people create insane or unreasonable content that caused performance to tank.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have frequently had reports of problems that only happen in the context of someone’s larger product with no repro case made available.</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I have frequently had people pass in garbage triangle mesh data, invalid normals, bad matrices, bad quaternions, and all manner of invalid and incorrect input data.</span></div>
</li>
</ul>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">So, as you can see, none of this is hypothetical. This is the life of a middleware developer. Let’s be frank, frequently when someone is licensing a piece of technology it’s because they don’t actually understand it that well themselves. They think that if they license this piece of middleware it will magically solve all of their problems. However, all middleware has integration costs and requires a good conceptual understanding of the underlying technology so that it is not abused and misused.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">If I was just writing something for myself and my personal colleagues, I probably would adopt only a few of these recommendations. However, if I was developing a piece of middleware going out to a wide audience that I had to support then I would probably do everything recommended here and more.</span></div>
<b style="font-weight: normal;"><br /><span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"></span></b>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: Arial; font-size: 15px; font-style: normal; font-variant: normal; font-weight: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">I hate to use the analogy but really I don’t see that I have any choice. You wouldn't have sex with a complete stranger without using a condom, you probably shouldn't let the users of your API do so without an equal degree of protection.</span></div>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline; white-space: pre-wrap;"></span>
<span style="font-family: Arial; font-size: 15px; font-weight: bold; line-height: 17.25px; white-space: pre-wrap;">Postscript</span><br />
<span style="font-family: Arial; font-size: 15px; font-weight: bold; line-height: 17.25px; white-space: pre-wrap;"><br /></span>
<span style="font-family: Arial;"><span style="font-size: 15px; line-height: 17.25px; white-space: pre-wrap;">I should probably address the elephant I left standing in the room here. Some people reading this are probably aware of middleware that I have been involved in and worked on. I would guess that those people are probably wondering how come the middleware I work on doesn't adopt hardly any of these recommendations. The answer to that is that it's not my middleware. I didn't create it and I don't own it. These recommendations are based on work that I have personally done in the past and, perhaps, some work I intend to do in the future. I hope this clarifies any questions you might have had on this point.</span></span><br />
<br />John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com12tag:blogger.com,1999:blog-23430315.post-8305297359891071582014-02-20T15:52:00.002-08:002014-02-25T08:57:09.369-08:00Narrative Clip : First Impressions<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipYzLts8r8-apWI4_Dzm8ismHhOqTHt3Pmtjj59PM30uEaO16mWiynbAIhGRh1vh9D7KWyccaSHpApAm0_SC_u8dj4lUMgle4AGvcVVhJCkerC6ldkFryPnOYj8WrYMovvfeRgEg/s1600/d4773690d22745f2a82c4fc40582f335.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipYzLts8r8-apWI4_Dzm8ismHhOqTHt3Pmtjj59PM30uEaO16mWiynbAIhGRh1vh9D7KWyccaSHpApAm0_SC_u8dj4lUMgle4AGvcVVhJCkerC6ldkFryPnOYj8WrYMovvfeRgEg/s1600/d4773690d22745f2a82c4fc40582f335.jpg" height="480" width="640" /></a></div>
<br />
I just received my <a href="http://getnarrative.com/">Narrative clip</a> a few days ago and I wanted to make a brief post about my first impressions. I was an early supporter of this Kickstarter campaign. In fact, it is the only Kickstarter I have every been involved in funding. I made my pledge back around October of 2012 and my original estimated shipping date was for March of 2013. I actually received my Narrative clip on February 18th, 2014, so a little less than a year late from the original quoted shipping date.<br />
<br />
The product was originally supposed to be called Memoto but, apparently, the company Motorola thought that was too close to their name and it was changed. Here is a link to the original <a href="https://www.kickstarter.com/projects/martinkallstrom/memoto-lifelogging-camera">Memoto kickstarter</a> fundraising campaign.<br />
<br />
When I first became interested in this device I wrote a blog article expressing my hopes for what the product might achieve. The main point of my article was to argue that if all the product did was collect thousands and thousands of pictures, then it would be largely useless and end up not being used. I argued that for the product to be useful it needs to be able to analyze and organize the data into a searchable narrative text that would describe the key activities in your life, and even be able to present it hierarchically. I wrote <a href="http://codesuppository.blogspot.com/2012_11_01_archive.html">this article</a> on November 12, 2012, a little over a year ago stating these goals. I became optimistic when the name of the product got changed to 'Narrative', as I was hopeful that they were going in that direction. I am sad to report that the initial version of the product simply collects a giant pile or photographs and really nothing more.<br />
<br />
At any rate, here are my initial impressions. Realize that my goal for this product is to have an efficient way to 'augment' my memory. I have a really, really, really bad memory so, for me, the hope of being able to augment it through technology is what attracted me to this device so strongly in the first place.<br />
<br />
First the good:<br />
<br />
<br />
<ul>
<li>The physical device is as advertised. It is a lightweight and rather innocuous clip-on which is easy to wear and quickly becomes unnoticeable. It could easily be mistaken for a clip-on MP3 player.</li>
<li>The pictures (sample above) are of reasonable quality. Most of them end up being of the floor and ceiling however. Learning how best to 'mount' it is something to work on. For example, during the work day, instead of clipping it on your shirt, you should set it on a flat surface and point it at your work-space. Likewise, when driving your car, maybe mount it somewhere on the dash so you don't just get a hundred pictures of your steering wheel. Some have suggested that it really needs more of a fish-eye lens since currently trying to 'frame' the picture through guesswork is a hit or miss affair. For women it's particularly problematic since they don't have flat chests and the camera is rarely pointing straight ahead. This is one big advantage Google Glass has since pictures it takes are always what you are currently looking at.</li>
<li>It does take pictures throughout the day, synchronizes, and uploads them to the cloud as promised. Some have complained about how long it takes to upload, but I have a very high speed internet connection so it hasn't bothered me that much yet.</li>
</ul>
<br />
<br />
Now for the bad, almost 100% of which has to do with the software. The good news is that the software can be improved. Meaning, you can collect the data and, at a later date, the software can still mine that data and present it to you in new and meaningful ways. On the other hand, they had an 'extra' year from the original promised ship date and they also received a lot of additional funding, so the extremely limited initial software can only be described as rather disappointing.<br />
<br />
<br />
<ul>
<li>I cannot comment on the quality and frequency of the GPS data as there is no way to review it. This is really surprising. I don't even know if the GPS is working, there is no way to tell at this time. </li>
<li>Connecting it to the computer seems to be a bit flaky, sometimes it recognizes the device, lots of other times it doesn't do anything.</li>
<li>I was amazed to see that there is no desktop or browser based application. The only way to review your data is on a mobile Android or IOS device which is awkward and slow. Someone else pointed out that they 'never promised this', and I guess that's probably true. Nevertheless, it's a really limited way to review the data.</li>
<li>The application you do get simply displays a thumbnail scroll-able window of a bunch of photographs. That's pretty much it. Since 99% of the pictures are just the floor, ceiling, or the steering wheel of your car, it quickly becomes clear that as an 'aid to memory' these photographs alone are going to be of questionable value. There is currently no way to see the GPS data corresponding with those pictures so good luck trying to figure out which floor or ceiling the picture represents.</li>
<li>One of the main features of a wearable lifelogging device is that it will be recording a constant snap-shot of GPS coordinates. With this data one could 'remember' everywhere they ever went! You could compute how far you drove, where you went, and even how far you walked or biked as well. Extracting from motion data whether you are walking, driving, or biking would be very straightforward to accomplish and computing distance is easy enough as well. If there was just a way to play back that GPS logged data on an interactive map. Currently there is no map view that I am aware of. This, to me, seems like really, really basic functionality that one would require in version 1.0 of a product like this. What good is a GPS lifelogging device if I can't get a view of everywhere I went? As I'm scrolling through the pictures on a timeline it should simultaneously be able to scroll through an annotated map, similar to how Google latitude used to work.</li>
<li>There is no face recognition or tagging.</li>
<li>There is no way to add notes or otherwise 'micro-journal'</li>
<li>There is no 'narrative'</li>
</ul>
So, as I said, this is just going to be 'first impressions'. I'm certainly going to give the device a fair try, and wear it for at least a month or two, letting it gather data into the cloud in the hopes that there will be software improvements down the road. Now that I have it, here are the features I would most like to see in order of priority<br />
<br />
<br />
<ol>
<li>A web-based or desktop based application rather than only having a mobile application</li>
<li>A map view that you can scrub to see everywhere you went throughout the day, similar to how Google latitude application used to work. It should, of course, also show the pictures simultaneously.</li>
<li>The ability to recognize 'locations' and be able to know concepts like 'home', and 'work' and the names of businesses that you frequent.</li>
<li>A breakdown of travel for each day; how far did you drive, bike, walk, and for how long.</li>
<li>A way to convert the day's activities into a text based journal describing things like, when you got up, when you went to bed, and the places you traveled.</li>
<li>A way to tag places and events, as well as add customized notes. For example, it might realize that you went to lunch at a particular restaurant, but you would have the ability to add notes to each event.</li>
<li>A way to tag faces and associate them with specific friends and, using face recognition, once you had tagged a face a couple of times it would be able to make the name association automatically. Facebook already has these kinds of features. Once this feature is incorporated, then the narrative text could include the names of people you had interacted with throughout the day.</li>
<li>A way to intuitively search the data, ideally with voice recognition, so that it could finally realize it's ultimate promise of augmenting memory. </li>
</ol>
John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com5tag:blogger.com,1999:blog-23430315.post-22195769768610411762014-01-04T07:19:00.001-08:002020-10-06T17:50:33.195-07:00How to Parse the Bitcoin BlockChain<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhh3w5xookIZqdgCqkr4yL-QvyUPiuuzOq6BZrddp2DBa3nY_kp9vzlpaNugmP6QJd9LniJbp9VZ7gxfXRjZq-EtF9zupMZNc-XT-evFLW_PXtTjD5uantFf-5_vKBxK3Qwzl02yg/s1600/blockchain.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhh3w5xookIZqdgCqkr4yL-QvyUPiuuzOq6BZrddp2DBa3nY_kp9vzlpaNugmP6QJd9LniJbp9VZ7gxfXRjZq-EtF9zupMZNc-XT-evFLW_PXtTjD5uantFf-5_vKBxK3Qwzl02yg/s640/blockchain.png" width="458" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
(Note as of January 8, 2014. A lot of people have been requesting a pre-built binary for the parser that has a command line interface. I just created one (for Windows 64bit machines only) and uploaded it. Here is the <a href="http://codesuppository.blogspot.com/2014/01/a-command-line-interface-for-blockchain.html">article documenting the tool with the download link</a>. You *can* build it directly from source code yourself if you want to <a href="https://code.google.com/p/blockchain">sync to the project</a>.)<br />
<div>
<br />
This article is going to describe in some detail how to parse the physical blockchain directly from your machine. The target audience for this article is developers, students, and researchers interested in writing their own software to directly parse the bitcoin blockchain data and interpret all of the transactions contained within. The scope of this article is to discuss how to read the blockchain data, how to interpret all inputs, outputs and transactions as well as to cover some nasty undocumented 'gotcha's' that you may encounter.<br />
<br />
I must give thanks for the article <a href="http://james.lab6.com/2012/01/12/bitcoin-285-bytes-that-changed-the-world/?c=917">'285 bytes that changed the world'</a> by James that helped get me started figuring all of this out.</div>
<div>
<br /></div>
<div>
My motivation for writing this article is because it took me many weeks looking in multiple locations and also a lot of debugging to figure all of this out and I want to save others all of that time and hassle.</div>
<div>
<br /></div>
<div>
Corresponding with this article is the open source <a href="http://codesuppository.blogspot.com/2013/07/bitcoin-code-snippets.html">bitcoin parser </a>that I have written and previously published on this site. If you don't want to write your own parser from scratch, you are welcome to take a look at mine which is documented <a href="http://codesuppository.blogspot.com/2013/07/bitcoin-code-snippets.html">here</a>.</div>
<div>
<br /></div>
<div>
The scope of this document, however, is *not* to discuss how blocks on the blockchain are mined or how transactions are validated. Since this article describes how the blockchain is physically stored on your machine this means that it refers to blocks which have already been mined and transactions which have all already been validated. Therefore you don't need to know the details about mining and transaction validation simply to interpret the raw transaction data itself.</div>
<div>
<br /></div>
<div>
To get a copy of the blockchain on your machine it is necessary to run the official bitcoin blockchain wallet application called '<a href="http://bitcoin.org/en/download">Bitcoin-QT</a>'. When this application starts up, it will begin to download the entire bitcoin blockchain to your hard drive which, as of January 4, 2014, is comprised of 106 roughly 128 megabyte files totaling over 14 gigabytes of data. It could take days for your machine to download the entire blockchain.</div>
<div>
<br /></div>
<div>
As of January 4, 2014 this 14 gigabytes of data contains:</div>
<div>
<br /></div>
<div>
278,619 : Blocks</div>
<div>
30,418,338 Transactions</div>
<div>
66,951,352 Transaction Inputs</div>
<div>
74,801,402 Transaction Outputs</div>
<div>
<br /></div>
<div>
The raw blockchain data files are stored in the following locations on your hard drive:</div>
<div>
<br /></div>
<div>
Linux: ~/.bitcoin/blocks</div>
<div>
MacOS: ~/Library/Application Support/Bitcoin/blocks</div>
<div>
Windows: %APPDATA%Bitcoin\blocks</div>
<div>
WinXP: C:\Documents and Settings\YourUserName\Application data\Bitcoin\blocks</div>
<div>
Win7/Win8/Vista: C:\Users\YourUserName\AppData\Roaming\Bitcoin\blocks</div>
<div>
<br /></div>
<div>
They will appear as a series of 128mb files blk00000.dat through blk00???.dat.</div>
<div>
<br />
The following documentation is a more detailed description of the labeled items in the image above. <br />
<br />
<h3>
Block Header : Section B1 : Magic ID : Uint32 (4 bytes)</h3>
</div>
<div>
<br /></div>
The first 4 bytes of a block in the blockchain file is a 32 bit header referred to as a 'magic id'. This is simply an identifier to let us know that we are at the beginning of a block. This magic ID will always be the number 0xD9B4BEF9. If you encounter a value other than that, you probably have a bug in your parser.<br />
<br />
It is important to note that it is possible that instead of finding the magic ID, you instead find a bunch of zero bytes. In my own copy of the blockchain I have encountered a case when parsing through a .dat file, where the header is missing. Instead there is a large block of zero bytes and then the blockchain picks up again later. I don't know why/how this occurs. <br />
<br />
It is also important to note that when you run the Bitcoin-QT client for the first time, it will download the entire blockchain to your hard drive, with no missing gaps between blocks or orphan blocks. However, as the bitcoin-QT is continuously running on your machine, it will introduce gaps into the .dat file as well as occasionally write out orphan blocks. Your parser will need to take this into account.<br />
<br />
<h3>
Block Header : Section B2 : Block Length : Uint32 (4 bytes)</h3>
<div>
The next 4 bytes of the blockheader contain the length of this block. Currently the maximum size of a single bitcoin block is 1mb; which is only big enough to support approximately 7 transactions per second, which quite frankly is rather disappointing. At a future time the block size may be increased but, if that were to occur, the already massive block chain would blow up out of control. This is a major issue for the bitcoin protocol. It cannot in any way support a massive number of transactions which could accommodate global commerce.</div>
<div>
<br /></div>
<div>
For today, however, you will never see a single block take up more than one megabyte of memory.</div>
<div>
<br /></div>
<div>
Another important note needs to be made here. The block-length is *not* equal to the amount of data in the block! Meaning, after you read the last entry in a block, that offset often can be, and will be, *less* than the total length of the block reported. Your parser must take this into account and seek the file pointer to wherever the next location in the file is indicated by the block length value.</div>
<div>
<br /></div>
<h3>
Block Header : Section B3 : Version Number : Uint32 (4 bytes)</h3>
<div>
The version number in the blockchain is always set to 1 in all cases. In the future this version number might get bumped and, if that were to occur, the layout of the data could, and likely would, change accordingly.</div>
<div>
<br /></div>
<h3>
Block Header : Section B4 : Previous Block Hash : 32 bytes</h3>
<div>
This entry contains a 32 byte hash of the 'previous' block. Think of this like a linked list. Each block in the blockchain points to the 'previous' block. It is important to note that the 'previous' block may not always be, well, the previous block! The blockchain can and sometimes will contain what are called <a href="https://blockchain.info/orphaned-blocks">'orphan' blocks</a>. <br />
<br />
Let me reiterate, you may encounter two blocks which both point to the same 'previous' block! How do you know which one to use? I explain that in more detail later in this article.</div>
<div>
<br /></div>
<div>
An orphan block naturally occurs when two miners solve the same block at roughly the same time. They are both technically 'valid' however, only one of the two blocks is accepted into the main chain. As you are scanning the blocks in the bitcoin blockchain, you will need to know how to link the blocks which are in the main chain by using this hash.</div>
<div>
<br /></div>
<div>
It is important to note that the hash of a block is *NOT* included in the block itself! It is a computed value. I will discuss in some detail later in this document how this hash is computed and how you go about patching up the linked list.</div>
<div>
<br />
A note about hashes. If you look at this hash in a memory dump, it will display in reverse order of what you see in the blockchain.info and blockexporer websites. For example, here's a link to <a href="https://blockchain.info/block/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f">block #0 (the genesis block) </a>on the Blockchain.info website. Note that it displays the hash as<br />
<br />
<a class="hash-link" href="https://blockchain.info/block-index/1/000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f" style="background-color: white; color: #0088cc; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 12px; line-height: 20px; text-decoration: none; white-space: nowrap;">000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f</a><br />
<br />
However, if you compute the hash and look at it in memory, it will show up as:<br />
<br />
6fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d6190000000000<br />
<br />
This is because the blockchain.info and blockexplorer websites display all of these values in 'big endian' format. This may be confusing at first when you are trying to debug your code.</div>
<h3>
Block Header : Section B5 : MerkleRoot : 32 bytes</h3>
<div>
The next part of the block is a 32 byte hash called the 'merkle root'. The merkle-root hash is not needed to parse the bitcoin transactions. It is an optimization feature used by the Bitcoin-QT application and some other applications to rapidly access transaction data without needing to have full access to the entire blockchain transaction history in memory. You can read a <a href="http://bitcoin.stackexchange.com/questions/10479/what-is-the-merkle-root">discussion on the topic here.</a></div>
<div>
<br /></div>
<h3>
Block Header : Section B6 : TimeStamp : uint32_t : 4 bytes</h3>
<div>
<br /></div>
<div>
This is the time stamp value indicating when this block was created. This is the only time value available in the bitcoin blockchain system and this also presents a huge problem. The timestamp is going to correspond to when the block was generated which is roughly once every ten minutes. At first you might think this is ok, since a ten minute resolution might be sufficient. However, it is not. Transactions are processed by miners in order of priority. The transactions contained in a block might, in fact, be *much older* than the block time itself! Transactions which do not include a fee could be many hours older than than the block they were finally included in.</div>
<div>
<br /></div>
<div>
I do not know why Satoshi did not decide to include a transaction time stamp. It's only 4 bytes, and it is a valuable piece of information that people really need access to. For now, all you can do is use the timestamp of the block and hope that's close enough for government work. And by government work, I mean close enough for tax and accounting purposes. Since the value of bitcoin is very volatile and can go up and down by massive amounts over a short period of time, clearly this coarse and inaccurate timing value is woefully inadequate.</div>
<div>
<br /></div>
<div>
This really needs to be fixed soon in my opinion.</div>
<div>
<br /></div>
<div>
This time value is in ANSI standard C 'time_t' format. You can learn more <a href="http://en.wikipedia.org/wiki/C_date_and_time_functions">about it here.</a></div>
<div>
<br />
<h3>
Block Header : Section B7 : Target Difficulty : uint32 : 4 bytes</h3>
</div>
<div>
This value represents the target difficulty for this block. This information can be ignored for the purposes of processing transactions. If you want to learn more about it, <a href="https://en.bitcoin.it/wiki/Difficulty">follow this link</a>.</div>
<div>
<br /></div>
<h3>
Block Header : Section B8 : nonce : uint32 : 4 bytes</h3>
<div>
The 'nonce' value is a random number value used as part of the mining process. It is not needed to be able to parse and interpret transaction data and may be ignored for our purposes here. If you want to know more about it's use and meaning, <a href="https://en.bitcoin.it/wiki/Nonce">follow this link.</a></div>
<div>
<br /></div>
<h3>
Block Header : Section B9 : TransactionCount : Variable Length Integer, 1, 3, 5, or 9 bytes</h3>
<div>
I'm not quite sure why Satoshi decided to use variable length integers. Maybe to save memory? So he could code 4 byte integers into 1 bytes? It's hard to say, because it offers very little value that I can see. Nevertheless, this is how he did it, and your parser will need to take it into account.</div>
<div>
<br /></div>
<div>
You can learn about how these variable length integers are coded by<a href="https://en.bitcoin.it/wiki/Protocol_specification#Variable_length_integer"> following this link</a>. </div>
<div>
<br /></div>
<div>
You can see an example implementation in this <a href="https://code.google.com/p/blockchain/source/browse/trunk/BlockChain.cpp">source code</a>. Search for the method: "<span style="background-color: white; font-family: "monaco" , "dejavu sans mono" , "bitstream vera sans mono" , "lucida console" , monospace; font-size: 12px; white-space: pre;">readVariableLengthInteger"</span><br />
<span style="background-color: white; font-family: "monaco" , "dejavu sans mono" , "bitstream vera sans mono" , "lucida console" , monospace; font-size: 12px; white-space: pre;"><br /></span>
<span style="background-color: white; font-family: "monaco" , "dejavu sans mono" , "bitstream vera sans mono" , "lucida console" , monospace; font-size: 12px; white-space: pre;"><br /></span></div>
<div>
<h3>
We are now at the portion of the block which contains the transactions. For each transaction you will find the following data:</h3>
</div>
<h3>
Transaction Section : T1 : Transaction Version Number : uint32 : 4 bytes</h3>
<div>
The transaction version number is currently always expected to be one or two. However, at some point during the lifetime of the block chain several blocks were accepted with some completely garbage transaction version numbers. To be safe, your parser should expect a version number of one or two but not fail if it sees something else.</div>
<div>
<br /></div>
<h3>
Transaction Section : T2 : The Number of Inputs : Variable Length Integer</h3>
<div>
This is a variable length integer which designates the number of inputs contained in this transaction.<br />
<br />
<h3>
Input Section : I1 : Transaction Hash : 32 bytes</h3>
</div>
<div>
This is the key magic piece needed to understand how inputs are handled. These 32 bytes represent a hash of some previous transaction in the block chain. <i>It is very important to note, this transaction hash is stored nowhere in the blockchain itself!!</i> It is a computed value. Later on in this article I will discuss in detail how this transaction hash is computed and how you can hook the inputs up correctly.</div>
<div>
<br /></div>
<h3>
Input Section : I2 : Transaction Index : uint32 : 4 bytes</h3>
<div>
Just as the previous value referred to a particular transaction this value says which output of that transaction comprises this input. If the transaction index is 0xFFFFFFFF then it means that this input refers to no previous output. You might wonder, who could such a thing occur. Well, it's because newly mined blocks get a block-reward of some value. Early on in the days of bitcoin this block-reward was equal to 50 bitcoins. Today is is worth 25 bitcoins, and down the road it will be worth half of that.</div>
<div>
<br /></div>
<div>
This is how new bitcoins are 'printed' or issued into the system. So, it is completely normal and expected that in a block the first input will refer to no previous output. The term often used for this is a 'coinbase' input.</div>
<div>
<br /></div>
<div>
Here is a link showing the single transaction of the <a href="http://blockexplorer.com/tx/4a5e1e4baab89f3a32518a88c31bc87f618f76673e2cc77ab2127b7afdeda33b">genesis block in the blockchain</a>. You will see that the input contains 50 bitcoins and refers to no previous output.</div>
<div>
<br /></div>
<div>
If the transaction index is non-zero, then it refers to a particular output in some previous transaction. It is important to note that inputs always completely spend all of a previous output.</div>
<div>
<br /></div>
<div>
This might seem confusing at first, but let me explain. Let's say you have a prior output corresponding to your address which has 10 bitcoins in it. Now, you decide that you want to send someone 2 bitcoins. You *must* spend all of the 10 bitcoins from the previous output. So, to send 2 bitcoins to someone you will create a transaction which has one input (the 10 bitcoins you previously had) and then two outputs. One of those outputs will contain 2 bitcoins for your friend and the other output will send the change of 8 bitcoins back to your address. Every set of transaction inputs fully spends some previous outputs in time.</div>
<div>
<br /></div>
<div>
If the total amount of bitcoins in the inputs is greater than the total number of bitcoins spent in the outputs, then whatever remains is known as a 'miner fee' and is awarded to the miner who created this block and included this transaction into the blockchain.</div>
<div>
<br /></div>
<h3>
Input Section : I3 : Script Length : Variable Length Integer</h3>
<div>
This is a variable length integer which contains the length of the input script.</div>
<div>
<br /></div>
<h3>
Input Section : I4 : Script Data : (length bytes)</h3>
<div>
This is the raw data which comprises the 'input script'. It is important to note, to parse all of the transactions in the block chain you do not need to know anything about the input script! You can simply ignore the contents of this data. The input-script combined with the output script, which we will cover in a moment, are used by the bitcoin system to validate transactions. However, we do not need to validate transactions. Since we are parsing the completed blockchain, we know that all of these transactions have already been validated.</div>
<div>
<br /></div>
<div>
If you want to learn more about what this script data contains and how it is used by the bitcoin protocol anyway, you can <a href="https://en.bitcoin.it/wiki/Script">read more about it here.</a></div>
<div>
<br /></div>
<h3>
Input Section : I5 : Sequence Number : uint32 : 4 bytes</h3>
<div>
The final piece of data for an input is the 'sequence number' which is currently always assigned to the value of 0xFFFFFFFF. You can ignore this field.</div>
<div>
<br /></div>
<h3>
Transaction Section : T3 : Output Count : Variable Length Integer</h3>
<div>
Once all of the inputs have been processed, the next thing you will encounter is the variable length integer which describes the number of outputs in this transaction.</div>
<div>
<br /></div>
<h3>
Output Section : O1 : Value : uint64 : 8 bytes</h3>
<div>
This is the most fun part of the whole thing. We finally get to the money! This 64 bit unsigned integer represents the output value measured in '<a href="https://en.bitcoin.it/wiki/Bitcoin">Satoshis</a>'. One Satoshi is equal to one hundred millionths of a bitcoin. So, for example, the first output in the genesis block will show up as 5,000,000,000 which is 50 bitcoins.<br />
<br />
<h3>
Output Section : O2 : Script Length : Variable Length Integer</h3>
</div>
<div>
This is a variable length integer which specifies the length of the output script.</div>
<div>
<br /></div>
<h3>
Output Section : O3 : Output Script : length bytes</h3>
<div>
This is where the magic sauce happens. Even though we don't have to execute the script to validate it, we do need to inspect the script to figure out where the output goes.</div>
<div>
<br /></div>
<div>
The public key in the output script will be stored in one of two forms. Either as the full 65 byte public key or as the 20 byte hash of the public key. Early on most public keys were stored as 65 bytes but, later, the vast majority of all transactions store the public key in the 20 byte form both to save memory and for added security.<br />
<br />
You can learn more about bitcoin public key addresses by looking at these links.<br />
<br />
ECDSA key (Elliptic Curve Digital Signature)<br />
<br />
<a href="https://en.bitcoin.it/wiki/ECDSA">https://en.bitcoin.it/wiki/ECDSA</a><br />
<br />
<a href="http://en.wikipedia.org/wiki/Elliptic_Curve_DSA">http://en.wikipedia.org/wiki/Elliptic_Curve_DSA</a><br />
<br />
You can decipher the public key for almost all of the outputs in one of the following few forms without necessarily having to write a full script interpreter. These usage patterns appear to be able to decode essentially every single valid public key address in the entire blockchain. There is a chance that scripts could change in form or be more complicated in the future, but for now these seem sufficient.<br />
<br />
<b>Format 1</b> : 67 byte long output script containing a full ECDSA 65 byte public key address.<br />
<br />
If you see an output script with a length of 67 bytes, the first byte is equal to 65 (indicating the length of the public key which follows), the next 65 bytes after the first byte are the public key, and the 67th byte (array index 66) is equal to 0xAC (172) which is the 'CHECKSIG' opcode.<br />
<br />
Script[0] =65 : Length of the public key to follow<br />
Script[1-65]=The public key data<br />
Script[66]=OP_CHECKSIG (0xAC)<br />
<br />
<b>Format 2 </b>: 66 byte long output script. Contains a 65 byte public key address.</div>
<div>
<br />
This script I believe is technically invalid. The 'length' field is missing, which is required for a valid script to execute. Nevertheless, this usage pattern does appear in the blockchain and is properly interpreted by a number of applications, so we need to accept it as well. This is in the same as format 1 except the first byte, the length field, is missing. In this use case the 66 byte long script begins with 65 bytes of the public key followed by the CHECKSIG opcode of 0xAC.<br />
<br />
Script[0-64]=The public key address<br />
Script[65]=OP_CHECKSIG (0xAC)<br />
<br />
<b>Fomat 3</b> : Script is 25 bytes long or more, contains a 20 byte public key hash address. This is the most common format of the vast majority of all output scripts.<br />
<br />
The early blocks in the block chain will be in format #1 and format #2 but, after a while, most of the scripts will be in this form. They will be 25 bytes or more long and begin with the following pattern.<br />
<br />
Script[0] = OP_DUP (0x76)<br />
Script[1] =OP_HASH160 (0xA9)<br />
Script[2] =20 (The length of the public key hash address which follows)<br />
Script[3-24] = The 20 byte public key address.<br />
<br />
<b>Format 4 </b>: Script is 5 bytes long and contains no public key.<br />
<br />
This script is in error. It is invalid and represents an unspendable address. It is documented here, however, because it does show up in the blockchain a number of times.<br />
<br />
Script[0] = OP_DUP (0x76)<br />
Script[1] = OP_HASH160 (0xA9)<br />
Script[2] =0 (A length of zero, not valid!)<br />
Script[3]=OP_EQUALVERIFY (0x88)<br />
Script[4]=OP_CHECKSIG (0xAC)<br />
<br />
<b>Format 5</b> : If the script doesn't readily match any of the previous patterns, then you can search for it by scanning the output script for the following pattern.<br />
<br />
Look for any place in the script where this pattern shows up:<br />
<br />
Script[0] = OP_DUP (0x76)<br />
Script[1] = OP_HASH160 (0xA9)<br />
Script[2] =20 (A length value equal to the size of a public key hash)<br />
Script[3-22]=The 20 byte public key address<br />
Script[23]=OP_EQUALVERIFY (0x88)<br />
Script[24]=OP_CHECKSIG (0xAC)<br />
<br />
If you see this, then you have identified the public key for this output script. In reality the vast majority of all output scripts will be in the form of format #3.<br />
<br />
<b><i>Finally, a note about how to interpret a public key.</i></b></div>
<div>
<br /></div>
<div>
As far as your code is concerned, you should only deal with the 20 byte public key hash. The question immediately arises about what to do if you encountered the 65 byte form, how do you convert it to the 20 byte public key version?</div>
<div>
<br /></div>
<div>
That process is documented in the following source code located here; <a href="https://code.google.com/p/blockchain/source/browse/trunk/BitcoinAddress.cpp">BitcoinAddress.cpp</a>, the header file <a href="https://code.google.com/p/blockchain/source/browse/trunk/BitcoinAddress.h">BitcoinAddress.h</a> is located here.</div>
<div>
<br /></div>
<div>
Look for the method 'bitcoinPublicKeyToAddress'. You will see that this method accepts a 65 byte ECDSA key and converts it into the 25 byte public key hash form. You might immediately wonder, wait..I thought the public key was 20 bytes? Well, it is in fact 20 bytes. However, as a matter of convention, bitcoin software adds a prefix header byte at the beginning and a 4 byte checksum at the end. These are derived/computed values and do not need to be stored.</div>
<div>
<br /></div>
<div>
Let's say you have a 20 byte public key but you need the full 25 byte form, then you can use the routine: 'bitcoinRIPEMD160ToAddress' to see how the header and checksum are added.</div>
<div>
<br /></div>
<div>
Of course, ultimately, probably what you really care about is seeing the public key address in the ASCII form that is everywhere on the Internet. </div>
<div>
<br /></div>
<div>
For example, here is a public key address: 115syJ2uJ4y5c2a6Ec6iWeeijZctBp955Q</div>
<div>
<br /></div>
<div>
You can use this public key address to send me delicious and tasty bitcoin tips for writing this long and painful article.</div>
<div>
<br /></div>
<div>
So, how would you convert this public key address to the binary form? In the source code provided above, you would call the routine 'bitcoinAsciiToAddress' which will convert from the ASCII Base58 format to the 25 byte form, and if you only need the 20 byte version you skip the first header byte and just use the 20 bytes following.</div>
<div>
<br /></div>
<div>
To convert the 25 byte address to the ASCII form we are used to seeing, you can use the routine 'bitcoinAddressToAscii'.</div>
<div>
<br /></div>
<div>
The source code above is in 'snippet' form. What this means is that you can use it only with the header and implementation CPP provided. It has no other dependencies and is wholly self contained.</div>
<div>
<br /></div>
<div>
This source code contains fairly extensive documentation on the topic of converting between 65 byte public key to the 20/25 byte RIPEMD160 public hash address form as well as into and out of Base58 ASCII.</div>
<div>
<br /></div>
<div>
And, once again..that tip address is: 115syJ2uJ4y5c2a6Ec6iWeeijZctBp955Q</div>
<div>
<br /></div>
<h3>
Transaction Section : Transaction Lock Time : uint32 : 4 bytes</h3>
<div>
The final data item at the end of each transaction is called the 'transaction lock time'. Currently this value is always set to zero, and we don't need to worry about it for now.</div>
<div>
<br /></div>
<h3>
Important notes once you have finished reading the last transaction! Please read!</h3>
Once we have consumed the final transaction, this brings us to the end of the logical block. However, and this is important to note, we will not necessarily bring us to the end of the physical block! The 'block length' specified at the beginning of this block may actually go beyond the end of the last transaction which was consumed. That is why it is important that you read the entire block into memory rather than just reading each transaction and expecting the file pointer to be in the correct location for the next block.<br />
<br />
Once you advance past 'block-length' worth of data, you should expect to either (A) reach the end of file, in which case you should try to open the next .dat file in sequence or (B) find a new header block magic id.<br />
<br />
However, there are some rare cases where after skipping ahead in the file to where the start of the next block header should be, instead you will encounter a block of zero bytes. I do not know why/how this can occur, but I assure you that it can occur and your parser will need to take this into account.<br />
<br />
If instead of finding the magic-ID header word you encounter a bunch of zero bytes, you will need to skip scan forward in the file until the magic-ID header appears and continue on the block chain from there.<br />
<br />
<h3>
First a Note about Hashes</h3>
<div>
<br /></div>
<div>
The rest of this article is going to discuss how you go about putting all of the blocks and transactions together so that you can interpret the flow of data. However, before I can do this, I need to provide some background explanation on the topic of hashes.</div>
<div>
<br /></div>
<div>
Here is the <a href="http://en.wikipedia.org/wiki/Hash_function">Wikipedia article on the topic of Hash functions.</a> Now, you can read all of that of course, but I'm going to just give a quick thumbnail sketch version here.</div>
<div>
<br /></div>
<div>
A hash is an operation on a set of variable length input data which produces an output signature that is typically much smaller in size. I suppose you could think of a social security number as a hash. Every social security number represents one person, but of course, that number doesn't tell you anything about the person himself.<br />
<br />
It is important to note that hash functions are almost always irreversible. While a hash function provides a unique id/signature for a set of input data, you cannot reconstruct the input data from it!</div>
<div>
<br /></div>
<div>
There are a wide variety of hash functions available. Hash functions typically take the input data and manipulate it in such a way that the output hash is extremely unique, it does this using a variety of binary operations which scramble the bits of the input data into the output hash signature.</div>
<div>
<br /></div>
<div>
The goal of a good hash function is that given two different sets of input data they rarely, if ever, produce the same hash value. If two sets of input data produce the same hash value that is what is referred to as a 'collision' and it is not desirable.</div>
<div>
<br /></div>
<div>
The input of a hash function is any variable length of binary data. The output is a fixed size hash value. The smaller the hash output, the more likely there is to be a collision. The bigger the hash value, the less likely a collision can occur.</div>
<div>
<br /></div>
<div>
So, let's take an example of the most simple hash imaginable. Let's say your hash function is simply the one byte exclusive-or value of the input data.</div>
<div>
<br /></div>
<div>
Two things. First, this isn't a very good hash function and, second, you would get a lot of collisions. In fact, even with a perfect hash function you would get one collision on average every 256 times.</div>
<div>
<br /></div>
<div>
Now, let's say we use a 16 bit hash. Now you can expect collisions to occur (two different sets of input data producing the same output hash) once every 65,536 times. Now consider 32 bit, now 64 bit, etc. etc.</div>
<div>
<br /></div>
<div>
Well, bitcoin uses a 256 bit hash called SHA256. Here is a link to a <a href="http://en.wikipedia.org/wiki/SHA256">Wikipedia article about it.</a> Here is a link to an implementation of it that I have provided. The header file <a href="http://sha256.hhttps//code.google.com/p/blockchain/source/browse/trunk/SHA256.h">SHA256.h</a> and the implementation <a href="https://code.google.com/p/blockchain/source/browse/trunk/SHA256.cpp">SHA256.cpp</a>. This implementation of SHA256 was created by Zilong Tan in 2011 and he released it under MIT license. He derived it from code originally written by Allan Saddi. All I did was repurpose the code for use in this bitcoin parser project. </div>
<div>
<br /></div>
<div>
You don't need to understand how SHA256 works to use it any more than you need to know how the arc-tangent function works on your calculator to use it. There is some concern as to whether or not SHA256 has been compromised by the NSA. SHA, which stands for 'secure hash algorithm',<a href="http://www.alertlogic.com/is-bitcoin%E2%80%99s-sha-256-bit-encryption-safe-from-nsa/"> was in fact developed by the NSA</a>. There are a lot of paranoid conspiracy theories out there on this topic. I can't really comment on them as it's not been my field of expertise. I take some solace from the fact that Edward Snowden is on the record as stating that '<a href="http://www.businessinsider.com/edward-snowden-email-encryption-works-against-the-nsa-2013-6">strong encryption works</a>', so let's just hope that he's right.</div>
<div>
<br /></div>
<div>
So, the theory goes that if SHA256 is a 'perfect' hash function then the chance of collision is so low that you don't even need to consider it as a possibility.</div>
<div>
<br /></div>
<div>
This is the magic of large numbers and cryptography in general. The human mind can grasp some numbers, like 1, or 10, or 100, or a thousand. The human mind can kind of grasp a million or even a billion in an abstract way. However, the human mind cannot grasp truly <a href="http://en.wikipedia.org/wiki/Large_numbers">large numbers</a>. Numbers on the order represented by these hashes are so large we cannot even begin to understand them.</div>
<div>
<br /></div>
<div>
So, apparently, the chance of a hash collision with the hash functions used by bitcoin are so remote that for all practical purposes we can ignore it. I tried to think of an analogy that people could kind of relate to and this is the best I came up with. The chance of guessing the private key for someone's bitcoin public address would be like winning the lottery every single day, for six days in a row, and then on the seventh day getting hit by lighting. That means you would have to win a major lottery, every single day, one after another, and, only then, once that happened, get struck by lighting. I hope this gives you some sense of how remote an event it would have to be for a hash collision to occur; it's considered for practical purposes such an unlikely event that we can think of it as virtually impossible.</div>
<div>
<br /></div>
<h3>
And now a discussion about HashMap or HashSet container classes</h3>
<div>
So, now that we have discussed the concept of how to compute a hash for a given set of input data, you need a way to do a 'look up'. A HashMap or HashSet can be thought of as a 'sparse' array or associative container.</div>
<div>
<br /></div>
<div>
Let's say you wanted to do an array look up of 10,000 items. No problem. You create an array with 10,000 elements and you index them.</div>
<div>
<br /></div>
<div>
However..what if your array lookup is a 256 bit hash? Yeah, not gonna happen. You would need an array so large it's absurd to even discuss it.</div>
<div>
<br /></div>
<div>
But, it turns out, that you *can* access it kind of like an array using what is called a '<a href="http://en.wikipedia.org/wiki/Hash_table">HashMap</a>'</div>
<div>
<br /></div>
<div>
The way it works is like this. You actually do create an array, one that is reasonable in size. Say we make an array that contains 8,192 elements. Now, when you want to 'find' an object based on a 256 bit hash (32 bytes), what you do is instead just compute a 13 bit hash of that 256 bit hash. (13 bits is from 0-8191).</div>
<div>
<br /></div>
<div>
You can now use this 13 bit hash an an array index and if there is nothing there, great, you add the object. If there is already something there (a collision occurs) then you add this element onto a linked list which begins at the array index. Collisions are expected to occur here, but they should happen with such a low frequency that it is acceptable. You need to strike a balance on the size of your hash-map array with the number of collisions you can live with. If your array is too small, then collision might happen frequently making the hash-map performance unacceptably slow. General purpose hash containers typically 'grow' the base hash array as collisions occur over time.</div>
<div>
<br /></div>
<div>
With a good hash which minimizes collisions, you can do extremely fast array like access even with massive keys.<br />
<br />
To do a lookup you generate the 13 bit hash, index the array, and then walk the linked list there to return any match found.</div>
<div>
<br /></div>
<div>
Now, a final note on this topic, the <a href="http://en.wikipedia.org/wiki/Standard_Template_Library">STL (standard template library) </a>comes with a built in <a href="http://www.sgi.com/tech/stl/hash_map.html">hash_map</a> container class. This sounds great, right? Why not just use it?</div>
<div>
<br /></div>
<div>
Well, you can, and I'm not saying your shouldn't. However, it is going to cause problems. Most general purpose container classes like the STL have to handle deletions as well as insertions. They do an enormous amount of heap memory allocation, often allocating memory for every single entry in the hash map. When you are dealing with data sets like bitcoin which currently contains on the order of 30 million transactions, that is going to create a problem. If you try to insert 30 million transaction hashes into an STL hash_map it's going to be slow and consume outrageous amounts of memory.</div>
<div>
<br /></div>
<div>
My recommendation is to implement a simple hash template that does fixed memory allocation and knows that you are only adding to the hash, never removing. In this way you can allocate the memory you need to store all transactions up front just one time, with no additional overhead. You can also associate a compressed index/transaction number id based on the location of this fixed sized array.</div>
<div>
<br /></div>
<div>
It's your call either way, but that is how I implemented the hash table lookups in my bitcion parser code. In the <a href="https://code.google.com/p/blockchain/source/browse/trunk/BlockChain.cpp">blockchain.cpp</a> file search for occurrences of the word 'SimpleHash' to see the implementation.<br />
<br />
<h3>
How to build the list of blocks in the main blockchain</h3>
</div>
<div>
Ok, now that you have a background in how hashes and hash maps work, we can discuss how to build the blockchain. Earlier in this article we explained how to read in a block from disk. </div>
<div>
<br /></div>
<div>
Once you have found the magic id, and read the block length, you must then read the block-prefix. </div>
<div>
<br /></div>
<div>
This is the 80 byte block prefix structure:<br />
<br />
struct BlockPrefix<br />
{<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>uint32_t<span class="Apple-tab-span" style="white-space: pre;"> </span>mVersion;<span class="Apple-tab-span" style="white-space: pre;"> </span>// The block version number.<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>uint8_t<span class="Apple-tab-span" style="white-space: pre;"> </span>mPreviousBlock[32];<span class="Apple-tab-span" style="white-space: pre;"> </span>// The 32 byte (256 bit) hash of the previous block in the blockchain<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>uint8_t<span class="Apple-tab-span" style="white-space: pre;"> </span>mMerkleRoot[32];<span class="Apple-tab-span" style="white-space: pre;"> </span>// The 32 bye merkle root hash<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>uint32_t<span class="Apple-tab-span" style="white-space: pre;"> </span>mTimeStamp;<span class="Apple-tab-span" style="white-space: pre;"> </span>// The block time stamp<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>uint32_t<span class="Apple-tab-span" style="white-space: pre;"> </span>mBits;<span class="Apple-tab-span" style="white-space: pre;"> </span>// The block bits field.<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>uint32_t<span class="Apple-tab-span" style="white-space: pre;"> </span>mNonce;<span class="Apple-tab-span" style="white-space: pre;"> </span>// The block random number 'nonce' field.<br />
};<br />
<br />
Here is an example of a struct that would hold a 256 bit (32 byte) SHA256 hash:<br />
<br />
struct Hash256<br />
{<br />
uint8_t hash[32];<br />
};<br />
<br />
Once you have read this in you must compute the block-hash.<br />
<br />
You do this by first computing the SHA256 hash of this 80 bytes of data. This results in an output 32 byte hash. You must then compute the SHA256 hash of this 32 bytes. This double hash is the 'block-hash'. Whenever any block refers to some 'previous' block hash it will be this computed value that it is referring to.<br />
<br />
Here is what this would look like in code:<br />
<br />
<br />
BlockPrefix prefix;<br />
fread(&prefix,sizeof(BlockPrefix),1,fph);<br />
Hash256 blockHash;<br />
computeSHA256(&prefix,sizeof(BlockPrefix),&blockHash);<br />
computeSHA256(&blockHash,sizeof(blockHash),&blockHash);<br />
<br />
Remember that if you print out the contents of the 32 byte 'blockhash' you must print it in reverse order to make it match up with the display in blockchain.info or blockexplorer<br />
<br />
Once you have computed this hash you need to add it to a hash map. The hash map entry should be something like the following structure:<br />
<br />
struct BlockHeader<br />
{<br />
FILE *mFph; // the file pointer where this block is located<br />
uint32_t mFileOffset; // the offset in the file where this block is located.<br />
Hash256 mPreviousBlockHash; // the previous block hash of this block.<br />
};<br />
<br />
You can then add this header to the hash map like this:<br />
<br />
mBlockHeaders[ blockHash ] = header;<br />
<br />
Once you have done something like this you will have created a hash map of every block found in the block chain.<br />
<br />
The next step is to actually build the main chain.<br />
<br />
You do this by starting with the *last* block in the block chain. Starting with the last block you will then do a hash lookup for the 'previous block'. You do this until there is no 'previous' block, which will be the first block, block #0, the genesis block in the blockchain. When you do this, you will likely end up skipping some of the block headers you previously scanned in. Those blocks you skipped are considered 'orphan' blocks and you can ignore them.<br />
<br />
Now you should have an ordered list of the bitcoin block chain from the first to the last. This hash map should not take up an enormous amount of memory, perhaps a few megabytes. With it, you can now seek to a specific file location and read the raw block data into memory and parse it in more detail from there.<br />
<br />
<h3>
Processing Transactions</h3>
</div>
<div>
<br /></div>
<div>
Ok, now that you have built a list of every block in the block chain you are ready to process the transactions. Just as you had to build a hash map for each of the blocks, you are going to have to build a hash map for each of the transactions, since each input refers to the transaction hash of the previous output.</div>
<div>
<br /></div>
<div>
The transaction hash is computed as follows:</div>
<div>
<br /></div>
<div>
Take the block of memory that encompases the entire transaction, from transaction version number to the transaction lock time. You take this chunk of memory and compute the double SHA256 hash just like you did for the block prefix hash. You can then add this hash into a hash-map so that when processing inputs which refer to previous outputs you can look them up. You can store in that hash-map the file seek location so that you can re-read it from disk if necessary.</div>
<div>
<br /></div>
<div>
For each block, you simply iterate all inputs and outputs to produce the details of the flow of money for each transaction.<br />
<br />
Since there are 30 million transactions to date, this is going to be a pretty damned big hash map. The blockchain also has this thing called a 'merkle-tree' which I believe is designed to help deal with this problem. I have not myself used the merkle-tree since I was happy just reading the whole damned thing into memory.<br />
<br />
In the future, some day, when there are billions of bitcoin transactions it would be impossible to hold all of this in memory and it would have to be replaced with a database.<br />
<br />
In fact, it's perfectly reasonable approach to just store all of this as data in an SQL database as you parse it. I'm sure a lot of people do that sort of thing.</div>
<div>
<br /></div>
<h3>
Processing Addresses</h3>
<div>
If you want to know the 'balance' of any public key address, you do this by collating all inputs and outputs of all transactions associated with that address. Once again having a hash map may be useful. You can build a hash-map indexed by the 20 byte public key hash address. The entry for each address would contain the list of transactions which have either inputs or outputs (or both) associated with that address.</div>
<div>
<br /></div>
<div>
To compute the balance you simply iterate through the inputs and outputs and compute the sum,.</div>
<div>
<br /></div>
<br />
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxzH9AZlPp0fHDMLfNf6Whq64RXjxoP7ocJJMld-gMtsazpBmU-UvN9r-b2Tk0ruP88mOr2qA7IZO-IDmswr2M86doPc3bWGXBjjTsDf4jegfuRkfDrEmiLDXUmK_6my7IcYe2fQ/s1600/bitcoindocumentationqr.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxzH9AZlPp0fHDMLfNf6Whq64RXjxoP7ocJJMld-gMtsazpBmU-UvN9r-b2Tk0ruP88mOr2qA7IZO-IDmswr2M86doPc3bWGXBjjTsDf4jegfuRkfDrEmiLDXUmK_6my7IcYe2fQ/s1600/bitcoindocumentationqr.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">TipAddress: 115syJ2uJ4y5c2a6Ec6iWeeijZctBp955Q<br />
If you find this documentation useful and it saves you some time on your project, please consider sending a bitcoin tip of any amount to this address. And, if you are Satoshi reading this, please send that tip out of the genesis block, or block #1 if that isn't possible. Thanks so much!</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAftNtD4ngxoNsGUuhnx2b2wp7rHZCCmh4b_mPze7pufva_YQ6ov95SOy-wd22oTbX6ig3ffyF7U_olioItsBjmgvEUgb1QGFUZ-GTPLQl4cSJzrVgRbRnWY0Ad9KI4JBhn2JJTg/s1600/Viking_03_1024x1024.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAftNtD4ngxoNsGUuhnx2b2wp7rHZCCmh4b_mPze7pufva_YQ6ov95SOy-wd22oTbX6ig3ffyF7U_olioItsBjmgvEUgb1QGFUZ-GTPLQl4cSJzrVgRbRnWY0Ad9KI4JBhn2JJTg/s1600/Viking_03_1024x1024.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="font-size: 13px;"><a href="http://laurenratcliff.com/collections/shop/">Lauren Ratcliff Designs accepts Bitcoin Payments!</a><br />
If you don't feel like sending a tip to me, then would you consider spending some of your bitcoin on some artwork? A lot of people who own bitcoin have never actually bought anything with it. Now is your chance. On my daughter's website, which accepts bitcoin via <a href="https://bitpay.com/">BitPay </a>and <a href="http://www.shopify.com/">Shopify </a>integration, you can buy some very inexpensive items like this sticker above. If you have never actually bought anything with bitcoin or gone through the <a href="https://bitpay.com/">BitPay </a>checkout process, I strongly recommend you give it a try!<br />
<div>
<br /></div>
</td></tr>
</tbody></table>
</div>
<div>
<br /></div>
<h3>
</h3>
John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com14tag:blogger.com,1999:blog-23430315.post-41794222136189933762013-12-14T10:34:00.003-08:002013-12-14T13:09:50.776-08:00An open source code snippet to build a view and projection matrix. Implements: D3DXMatrixPerspectiveFovLH, D3DXMatrixOrthoLH, and D3DXMatrixLookAtLH<div class="blockchain-btn" data-address="1BsD7og5WKuh8b6eh6CGYAMM9EE8H6wD6d" data-shared="false" style="font-size: 16px; margin: 0 auto; width: 300px;">
<div class="blockchain stage-begin">
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZHzWjGSH0YbneabqujS_NAaezjRyLi_-9sEw09Ujg-mAIVlYSEVj9MF6CeAzUmhwjY3nNC9BVCVEgxZMapKDYcAFnWHiwZytbEZJroA4guhE6QAzXvWJk1ASfhKClXqEKNV5xhQ/s1600/pic.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZHzWjGSH0YbneabqujS_NAaezjRyLi_-9sEw09Ujg-mAIVlYSEVj9MF6CeAzUmhwjY3nNC9BVCVEgxZMapKDYcAFnWHiwZytbEZJroA4guhE6QAzXvWJk1ASfhKClXqEKNV5xhQ/s400/pic.png" width="300" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Artwork by <a href="http://laurenratcliff.com/collections/shop">Lauren Ratcliff</a></td></tr>
</tbody></table>
<br />
Here is a link to the code snippet: <a href="https://code.google.com/p/codesuppository/source/browse/trunk/old_snippets/ViewProj.h">ViewProj.h</a><br />
I have a pet-peeve that Microsoft doesn't provide source code to D3DX. It's very annoying to call various utility functions and not be able to step through the source code; especially for basic math and utility functions.<br />
<br />
One very common thing all 3d applications need is the ability to build a view and projection matrix. Go ahead and search the Internet on the topic. You will find thousands of articles and implementations however, every single one of them, will be specific to either OpenGL or D3D and/or depend on custom vector and matrix math libraries.<br />
<br />
I just find this to be really annoying. So, today, I'm providing a vanilla source code implementation of the three common methods used to build a view and projection matrix. It is just a handful of inline function calls in a single header file. It does not depend on or use any vector or matrix libraries.<br />
<br />
To use the routines you simply pass in a float pointer. If your code base already has it's own vector/matrix routines then simply cast them to a float pointer. <br />
<br />
You can also use the source code snippet as a learning exercise. <br />
<br />
If you find this source code snippet useful, please feel free to send me some bitcoin to the following tip-jar address:<br />
<br />
<a href="https://blockchain.info/address/1BsD7og5WKuh8b6eh6CGYAMM9EE8H6wD6d">1BsD7og5WKuh8b6eh6CGYAMM9EE8H6wD6d</a><br />
<br />
You can also use the QR code embedded in the image above. The artwork was done by my daughter Lauren. Here is a link to her <a href="http://laurenratcliff.com/pages/showreel">website</a>, where you can buy artists prints and other items. She accepts bitcoin as well.</div>
</div>
John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com0tag:blogger.com,1999:blog-23430315.post-85490828595765658802013-07-16T15:16:00.002-07:002013-07-17T08:08:26.709-07:00Ubuntu<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi33FqMOvNB7V0G_5u83_NTpx0SR29hkSyv9HMoz1b3BoTa-a5A1XsvnI6KSVzCGEqN9i2zae4fQkSlPjeo6TmtfVVDA8lQ41G4eJIsrHz9YcJUsasSRmldupPGhrrH-mRUj12ySw/s1600/ubuntu-1024x819.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="318" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi33FqMOvNB7V0G_5u83_NTpx0SR29hkSyv9HMoz1b3BoTa-a5A1XsvnI6KSVzCGEqN9i2zae4fQkSlPjeo6TmtfVVDA8lQ41G4eJIsrHz9YcJUsasSRmldupPGhrrH-mRUj12ySw/s400/ubuntu-1024x819.png" width="400" /></a></div>
<br />
I'm not really much of a Linux guy, but I do enjoy doing open-source development with small code snippets. Today I installed Ubuntu Desktop as a virtual machine on my main development box.<br />
<br />
Going forward, I'm going to get in the habit of always uploading valid Linux builds for all of my open source projects and code snippets. <br />
<br />
That was really all I had to say, hopefully I'll update some projects soon.<br />
<br />
I am proud to say that I just revised the bitcoin blockchain parser file to build with Linux! There were a handful of minor build issues. Going forward into the future I always intend to provide a working Linux version of my code snippets on this site; unless they are for some reason Windows specific, which usually will not be the case.<br />
<br />
Here's the link to the <a href="https://code.google.com/p/blockchain/">bitcoin blockchain project</a>; with the Linux make file.<br />
<br />
Here's a link to the <a href="http://codesuppository.blogspot.com/2013/07/a-bitcoin-blockchain-parser-as-single.html">original post</a> discussing it.John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com0tag:blogger.com,1999:blog-23430315.post-53822456424292481482013-07-11T14:40:00.001-07:002013-07-11T14:40:43.408-07:00User defined Bit Filtering Class<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqRjC4ZDx8W7arb1KYIElcTWWabthAXOzjGVH1LTnY88U3wZp6bW0FM4hYt5hLgS_WbLxeTIOBvKyzKbJr3fmQVnEPN0IaixBDW2cGKPg-niklLil22V_lns851ecJ-r-rZnpvNg/s1600/filtering.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqRjC4ZDx8W7arb1KYIElcTWWabthAXOzjGVH1LTnY88U3wZp6bW0FM4hYt5hLgS_WbLxeTIOBvKyzKbJr3fmQVnEPN0IaixBDW2cGKPg-niklLil22V_lns851ecJ-r-rZnpvNg/s400/filtering.jpg" width="266" /></a></div>
<br />
Today I am providing a small code snippet which I have found to be convenient.<br />
<br />
Here is the header file: <a href="https://code.google.com/p/filterbits/source/browse/trunk/FilterBits.h">FilterBits.h</a><br />
Here is the implementation: <a href="https://code.google.com/p/filterbits/source/browse/trunk/FilterBits.cpp">FilterBits.cpp</a><br />
And if your compiler does not provide 'stdint.h' here is a link to a version of that include file: <a href="https://code.google.com/p/filterbits/source/browse/trunk/stdint.h">stdint.h</a><br />
<br />It is an extremely common use case in software development where you might want to define a collection of groups by name and give each group/property a representation as a bit in a bitmask integer value.<br />
<br />
These flags are typically used to represent properties on an object, and one object might have multiple properties. You might have a set of bits which define what 'type' of object this is, and yet more bits that describe other properties of that object (friendly, enemy, or inside, outside, etc. etc.)<br />
<br />
Often times you will have a set of bits which define the properties of an object and then another set of bits which describe which properties of another type of object this object reacts to. This is often used in 'collision filtering' where you take two objects and see if the filter bits match the type bits of another object. If they do, then you will treat the test as 'true' otherwise 'false'.<br />
<br />
More complex boolean operations can be performed, but this is the common use case.<br />
<br />
Now, here's the problem this code snippet is trying to solve. In almost all cases where a project needs to have a set of bitfields, representing properties on an object, those bitfields are hard-coded and predefined in the source code. Usually with an enumerated type or some defines.<br />
<br />
Example:<br />
<br />
enum ObjectType<br />
{<br />
OT_PLAYER = (1<<0 p=""> OT_TERRAIN = (1<<1 p=""> OT_NPC = (1<<2 p="">};<br />
<br />
However, if you want to make a general purpose property system, that is not hard coded, this doesn't really map very well.<br />
<br />
This code snippet lets an artist/designer decide which 'object types' there are. It's completely up to them, so long as they don't create more than the maximum number of 64 unique types, it is fine.<br />
<br />
The provided sample code will parse the input types and convert it to a bit-pattern that you can use in C++.<br />
<br />
Personally, I have found this to be a useful and valuable tool.<br />
<br />
There are two reserved 'types'. They are the type 'all' and 'none'; meaning exactly what you think they would mean.<br />
<br />
For example, the artist/designer might define a string like:<br />
<br />
Player,Enemy=All<br />
<br />
Which could be interpreted as meaning this is a player, and an enemy, that interacts with 'all' other types of objects.<br />
<br />
This code converts the input string in the form of: "a,b,c=d,f,g" into a 64 bit source bit pattern and a corresponding 64 bit destination bit pattern.<br />
<br />
The returned bit patterns are guaranteed to be persistent, that the the application can cache them for future use and does not need to ever have them regenerated.<br />
<br />
If you don't need 64 bits and only 32, then there are accessor methods which will return the results as two 32 bit numbers instead.<br />
<br />
The intended use of this snippet is so you can allow artists/designers to create a set of arbitrary ASCII strings which represent the properties pair, defining type and interaction, in some kind of editing tool. At run-time your code just passes in the string on startup and gets back the run-time binary code which is all you use from that point forward (i.e. don't convert the strings to bitmasks every time you want to use them.)<br />
<br />
<br /></2></1></0>John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com1tag:blogger.com,1999:blog-23430315.post-53273079464191355062013-06-15T12:14:00.001-07:002013-06-15T12:20:49.013-07:00Box Approximation : A potentially useless piece of source code<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1w7LsRAaUu6b2lnnK9D-U3pz-n6vfnlHI8A8dq1j20aNDcOqJIUy6SoQkfvCONZXfMvtH8isRGk2L1NOlICpi5yZEtnyNhEmI8rKXZkkqcDIZdjoySIQcTvKB4NQPRplfD32UJA/s1600/bunny.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="254" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1w7LsRAaUu6b2lnnK9D-U3pz-n6vfnlHI8A8dq1j20aNDcOqJIUy6SoQkfvCONZXfMvtH8isRGk2L1NOlICpi5yZEtnyNhEmI8rKXZkkqcDIZdjoySIQcTvKB4NQPRplfD32UJA/s640/bunny.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_eRcg_chFA3CGEe7cVOPfpvlMQRInWNpxxwutOEj4OPq1B9buSWyvws3sP6C9Wp1c8XZy1kdkHLsodPZ6UB7tjZDssf6jp4kLCRg17Wh2UHRV6WmKRJziBdWA7AwdanXKwUoEPw/s1600/chair.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="254" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_eRcg_chFA3CGEe7cVOPfpvlMQRInWNpxxwutOEj4OPq1B9buSWyvws3sP6C9Wp1c8XZy1kdkHLsodPZ6UB7tjZDssf6jp4kLCRg17Wh2UHRV6WmKRJziBdWA7AwdanXKwUoEPw/s640/chair.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaOuMTA6_KD6H8x3lge1AW7WA_hAgaw0pKOQsnafc5i08npoSHcRFxcHL00_QZlfhTMVstqtZrteEBH7v7FWxLHDCSxElJRLzrwgN5F3OP3YiLK8-0-xPifm_DE-YPB0Yt-1KxIg/s1600/chesstable.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="256" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaOuMTA6_KD6H8x3lge1AW7WA_hAgaw0pKOQsnafc5i08npoSHcRFxcHL00_QZlfhTMVstqtZrteEBH7v7FWxLHDCSxElJRLzrwgN5F3OP3YiLK8-0-xPifm_DE-YPB0Yt-1KxIg/s640/chesstable.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2nLC1pByAMdNNRnqT7vKugtYspN2aLOYKxG5x7mtqhwhdU20dCewFP2xxBXjfCSQFqSb0qE9peyzTQ-NDDn9EnnGT9XrbYcdEt3sH54dvbFKjs2Agnh1oWHU9R5ewU_INDKVAnA/s1600/hornbug.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="252" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2nLC1pByAMdNNRnqT7vKugtYspN2aLOYKxG5x7mtqhwhdU20dCewFP2xxBXjfCSQFqSb0qE9peyzTQ-NDDn9EnnGT9XrbYcdEt3sH54dvbFKjs2Agnh1oWHU9R5ewU_INDKVAnA/s640/hornbug.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1NM-m3LvrEb_H0Q4CP7YBBartW_mNIpe502boPTakg1Ft5wKcGb6QEZ5WuKmXNIB1iSz1wXAThQnFWTsBIE60qqq6E2abC66glW8jUhT7eKB751krW2nO9PWP3ZkucM-VAH6nNg/s1600/image1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="256" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1NM-m3LvrEb_H0Q4CP7YBBartW_mNIpe502boPTakg1Ft5wKcGb6QEZ5WuKmXNIB1iSz1wXAThQnFWTsBIE60qqq6E2abC66glW8jUhT7eKB751krW2nO9PWP3ZkucM-VAH6nNg/s640/image1.png" width="640" /></a></div>
<br />
<br />
Yesterday I wrote a piece of experimental code that I don't know if it has any practical useful value. However, I did spend a little bit of time writing it and since there is a chance that it may solve some use case that I am not aware of I'm posting it here anyway. I am at least sure that the code is useful as a learning tool.<br />
<br />
Here is what I was trying to do. I wanted to see if I could approximate game level geometry as a set of box primitives instead of triangles. Now that I have done it, I'm not sure if I even have any use case for the method. That said, perhaps somebody else has a use case where they have a giant data set of boxes and want to optimize them or maybe they have a triangle soup and know of some value in representing them as a crude approximation based on boxes. <br />
<br />
At any rate, here is how the algorithm works.<br />
<br />
The first step is to convert the triangle mesh into a bunch of boxes. This happens automatically when you generate an axis-aligned-bounding-volume tree (AABB) for a bunch of triangles. I have previously uploaded <a href="http://codesuppository.blogspot.com/2012/06/raycast-mesh-bug-fixes.html">sample code</a> demonstrating how to build an AABB tree for a triangle soup, suitable to be used for raycasting. <br />
<br />
So, I revised this demo to output just the set of boxes produced. However, once you do this, you end up with an outrageous number of boxes because the algorithm is recursive. You can easily end up with hundreds of thousands, even millions, of boxes based on how the AABB performed recursion on the source data.<br />
<br />
So, this is the part of the algorithm that I found kind of interesting. The question was, how could I quickly combine all of those boxes back together to form larger optimal boxes?<br />
<br />
In the source code included with this post, you will find an interesting code snippet called 'BoxApprox.cpp'. This tiny bit of source code takes as input a giant collection of boxes and will produce, as output, the coalesced minimum box representation of the original.<br />
<br />
Here is how the algorithm works.<br />
<br />
Step #1 all of the boxes are converted into a quantized form (meaning floating point values are converted to fixed precision integers). In the source code included I use 20 bits of resolution for each axis. The sample code uses 64 bit integers to store the quantized address; so 20 bits for X, 20 bits for Y, and 20 bits for Z; taking up a total of 60 bits to represent any hashed address in the space of the volume.<br />
<br />
Once all of the boxes have been converted into quantized integer values; they are added to a hash-map. Using the hash map you can now find the upper left hand corner of any box in the list just by doing a very fast hash look up. Each box is represented as it's address and it's quantized size value.<br />
<br />
In step #2, we want to combine all boxes into larger boxes if this can be done. The way you do this is that you check to see if a box exists at the neighboring corner of a source box. If the neighbor box is the same size on two dimensions, then it can be combined into the original box. To find the neighbor you simple take the quanitzed X,Y,Z value and add the DX, DY, or DZ to each axis and form a new address for a fast hash lookup. <br />
<br />
Now, if you simply run this algorithm on the input boxes combing them until you can no longer combine two boxes any more, it works, and it is fast, but you will not get an optimal solution. Since the source boxes got combined in arbitrary order, you will get long thin slivers of boxes.<br />
<br />
So, the way to fix this is is that you need to create a priority list of all boxes based on their volume. <br />
<br />
Then, each time you want to consider a box to be combined, you can always start with the lowest volume box in the list.<br />
<br />
Each time two boxes get combined, both of the entries for the old boxes are removed from the priority queue and the new box, with its newly computed volume, is re-inserted into the priority queue.<br />
<br />
Wash, rinse, repeat, until you no longer can combine boxes any more and you are done.<br />
<br />
The sample BoxApprox.cpp also contains code which demonstrates how to take a list of boxes and write them out as an indexed triangle mesh. This is included here just for debugging purposes.<br />
<br />
To use the tool you run the executable 'BoxApprox.exe'.<br />
<br />
You pass in two command line arguments, the first is the name of a wavefront OBJ file containing a triangle mesh to approximate with boxes, and the second is the recursion depth to use when building the AABB tree. The larger the recursion depth, the longer the operation will take to perform. Try not to use a recursion value much larger than 15 or you will tie up your computer for a long time. After it finishes creating the box approximation the tool will write out a wavefront OBJ file with the same name but prefixed with 'boxes_'.<br />
<br />
Like I said at the beginning of this article. I have no idea if this code or algorithm has any practical purpose. It seems a little backwards to take an efficient triangle mesh representation and convert it to a crude approximation of box primitives. The only thing is, sometimes boxes can be a useful representation and, hey, maybe it's just something you want to do for artistic reasons.<br />
<br />
At any rate, here it is. Even if the actual algorithm is not useful, I'm pretty sure the code is a good educational demonstration of how to do quantization of floating point numbers, and how to use STL hash-maps and a few other things.<br />
<br />
If you found this code useful in any way, or this blog in general, I am happy to accept tips in the form of <a href="http://bitcoin.org/en/">Bitcoin</a> payment to this address; all tips are appreciated. Even just a fraction of a bitcoin is real money these days.<br />
<br />
Here is the link to the<a href="http://www.duatiu.com/BoxApprox.zip"> zipped up source code and demo files.</a><br />
<h3>
BitCoin Tipping Address: "18BQAoEnbEEiM6dVhuJZFHwjt1gLdkDJwp"</h3>
<br />John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com3tag:blogger.com,1999:blog-23430315.post-77594680905214136422013-04-03T10:30:00.001-07:002013-04-03T14:28:02.001-07:00Some clarifications on my previous post about young people trying to 'get into' the game industry..<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhArZuS5e-JvSBXGi-J6RQCF9m2J5MqxZCtf7V3AAqQPV06drICjT7ZsJMTxy-GIs-AWRamblLauyL0-RXdtv9KfCQauv5k88dR47QNl_k_Z-ALBiWL1L4U0IaxOSvA8OdKqOEkUw/s1600/i_love_clarification_mug-p168838732436870017enw9p_4001.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhArZuS5e-JvSBXGi-J6RQCF9m2J5MqxZCtf7V3AAqQPV06drICjT7ZsJMTxy-GIs-AWRamblLauyL0-RXdtv9KfCQauv5k88dR47QNl_k_Z-ALBiWL1L4U0IaxOSvA8OdKqOEkUw/s400/i_love_clarification_mug-p168838732436870017enw9p_4001.jpg" width="400" /></a></div>
<br />
My previous post titled <a href="http://codesuppository.blogspot.com/2013/04/so-your-teenager-tells-you-they-want-to.html">"So your teenager tells you they want to 'make video games for a living'.."</a> has gotten a huge response. I wanted to take a moment to briefly address some of the criticism it has received.<br />
<br />
So far, almost all of my friends who actually work in the game industry have given me strong positive feedback, but, there have been a lot of people upset about it too.<br />
<br />
My original post was quite lengthy and had some <a href="http://cbloomrants.blogspot.com/">Charles Bloom</a> rant-like elements to it. That's a good thing, a post which is full of qualifications and half-expressed opinions can feel a bit wishy-washy. The fact that it has gotten a strong response and spawned a lot of discussion is a good thing too. Nevertheless, I want to address some very specific criticism I have received.<br />
<br />
<h4>
<i>(1) People take issue with my claim that you have to be a really great artist to work as an artist in the game industry.</i> </h4>
<br />
I really don't know what to say about this. Every single one of my friends who is a professional artist in the game industry strongly and enthusiastically supported this view. The level and quality of artwork in video games today is incredibly high. I really don't see much opportunity for people working as a professional artist in the game industry without being exceptionally talented.<br />
<br />
<h4>
<i>(2) People take issue with my claim that you must be 'born' with artistic talent to become a great professional artist.</i></h4>
<br />
I really don't know what to say about this either. In my experience, and by the way <a href="http://www.duatiu.com/jratcliff/gallery/index.html">I am an artist myself</a> though not good enough to be a professional one, I have never met a great artist who did not have innate talent. To me this is kind of like saying 'anyone can learn to be a great musician' or 'anyone can learn to be a great singer'. I think most people would consider that an absurd statement and to me, it is equally absurd to say 'anyone can learn to be a great artist'.<br />
<br />
I'm sorry, I just really do not buy this argument at all.<br />
<br />
<h4>
<i>(3) Many people got upset by my statements about the amount of math necessary to become a computer programmer in the game industry. Even John Carmack took issue with that statement in a comment on the original post. I realize that I didn't explain this carefully enough, so I will now.</i></h4>
<br />
To begin with, the core of my argument is that you should not get a degree/certificate from a 'gaming college'. I have said this, as well as many others, that not only are most of these institutions running a scam, preying on the hopes and dreams of young people, but most importantly they terribly limit your opportunities.<br />
<br />
My argument is that if you aren't good enough to be accepted to a respected university for a degree program as a software engineer, why would you assume you have the necessary skills to work in the video game industry?<br />
<br />
My argument is that you should pursue a college degree in computer science at a respected university. By doing so, you open yourself up to the opportunity to get a job in a lot of fields with any number of technology companies. The minimum math requirement to get a degree in computer science at most schools is three semesters of calculus, differential equations, matrix and linear algebra.<br />
<br />
Now, you may not need all of that math to actually program games. But you sure need all of that math to get a computer science degree. I think this may be the distinction people were missing in my original post.<br />
<br />
Yes, lots of things in computer games today require a solid math background. And, yes, there is also a lot of work in computer games which do not require great math skills. But, I assure you, that a degree in computer science from a respected University is going to require that you to take these classes.<br />
<br />
The audience for my original article was to parents with teenagers wanting to pursue 'video games' as a career. My point was, and remains, that if your child does not have an aptitude for math then it's probably a strong indication they are not going to be successful on a path as a software engineer. <br />
<br />
I didn't mean to imply that your kid must be a 'math genius' literally. What I meant is that only a small percentage of high school students are interested in and take advanced math classes. Compared to the rest of the student body these kids are the 'brains', the 'smart kids'. The mere fact that you take a calculus class in high school puts you in some pretty rarefied company. You are probably on the math team and probably have a deep love and interest in math.<br />
<br />
Most importantly to my original point, having that level of math in high-school is generally considered a <b>prerequisite </b>to pursue a degree in computer science. When I went to school the math requirement was the first thing that weeded out most engineering students.<br />
<br />
So, what I was trying to get at is that if your teenager expresses an interest in programming computer games but shows no aptitude for math, that should probably rule them out for pursuing a degree in computer science.<br />
<br />
A software engineer is a problem solver. That is what they are driven to do. It ultimately doesn't matter so much whether those problems are in computer games, or finance, or robotics, or any one of hundreds of fields. Personally I have worked on software in a lot of different fields, including doing cardiovascular research at St. Louis University hospital while, at the same time, I was writing a computer game for Electronic Arts.<br />
<br />
In my experience, the standards for being hired as a software engineer (in a salaried position) at a game company are really, really, high.<br />
<br />
I've done a bunch of hiring in my life and I know the kind of standards we expect candidates to meet. More to the point, I have gone on interviews at game companies where they wouldn't even consider hiring me either.<br />
<br />
I want to be very clear here. I am not that brilliant of a guy. I am certainly not a genius or anything like that. What I am is a very hard worker. I have a very strong work ethic and I am really good at computer programming and debugging. I am largely self taught since most everything we do in the game industry was invented long after I left college.<br />
<br />
I should also point out that I do not, personally, have a four year college degree. I have an associate of arts degree and then I attended college at the University of Missouri Rolla to pursue a degree in computer science.<br />
<br />
At the time that I attended school we still used punch-cards and programmed in FORTRAN and COBOL. The C language wasn't even a consideration back then.<br />
<br />
Due to financial reasons I was never able to finish my degree and I paid for that heavily.<br />
<br />
My employment options were very limited. I could not get a job at any one of hundreds of technology companies which require a bachelors degree to get in the door.<br />
<br />
I made far less money in salary than I could have. I worked for a couple of years writing games and educational software, largely in assembly language, for a $12,000 a year salary! At the same exact time (this is 1982) my friends who got their four year degree from UMR in computer science were going to work for McDonnell Douglas, Lockheed, and other companies with starting salaries of nearly $50,000 a year! It took me many years go get up to that level.<br />
<br />
And, over the years, I have seen many self-taught computer programmers, as talented as they may have been, struggle with employment opportunities and lower salaries.<br />
<br />
So, my advice is all born out of personal experience.<br />
<br />
Can you be a brilliant computer programmer without ever going to school? Absolutely. Can you write incredible software without knowing a great deal of math? Without a doubt.<br />
<br />
However....you are digging yourself a very long and deep hole if you do so.<br />
<br />
As a parent, do you want to steer your child towards a career path which affords the most opportunity? Then, my advice is, and remains without apology, that you should dissuade them from pursuing a limited education at a 'gaming institute' and instead encourage them to pursue a degree in computer science from a reputable University. That will afford them them best salary and the widest array of employment opportunities in the future. And, if they do not have the aptitude, interest, or desire to pursue a degree in computer science then I would argue, for sure, getting a lower degree with a lower standards and quality education from a 'gaming school' is a complete and utter waste of time and money.<br />
<br />
<h4>
<i>(4) One thing that really annoyed people is that I talked about how there were 'only two jobs' in the game industry and that is simply not true.</i></h4>
This is another case where I didn't parse things carefully enough. Yes, there are lots of other jobs in the game industry other than simply artist or programmer. However, many of those jobs are low paying and, therefore, not viable as a career choice (which was the focus of my original article).<br />
<br />
Let's put it this way... because I have been in the game industry for so many years, I am constantly flooded with notices from headhunters. I daily see them looking for artists and programmers with varying levels of experience. I can barely remember them ever putting out an open casting call for a game designer. I'm sure it happens, but not that often. There are a handful of celebrity game designers but it's hardly the sort of thing you go get a dedicated college degree in and then fight off the six figure job offers that come pouring in. <br />
<br />
The problem with these other jobs is that for every one job opening there are probably hundreds and hundreds of people who would love to get it. People want to get into the game industry so badly that they will work for free just for the chance.<br />
<br />
I understand how much young people love games and romanticize the idea of working for a game company. But, as a parent, is this really the career path you want to direct your child towards?<br />
<br />
Other jobs in the game industry, such as in management and marketing, are certainly available, but a conventional business degree is all that is necessary to pursue those. For game design, a liberal arts program with some computer classes and game theory are all good, but don't set yourself up for failure by limiting your career choices.<br />
<br />
My advice all stays the same. Don't limit yourself by attending a 'game university'. Get a degree where you can find opportunities in many fields and industries. <br />
<br />
But, if you pursue a path where the single and only outcome is getting one of the few, rare, jobs open for a game designer in the game industry, you are setting yourself up for a major disappointment. The chances are really quite low.<br />
<br />
<h4>
<i>(5) Another critique is that I didn't address all of the opportunities available in the game industry as a whole.</i></h4>
This is true because my article was specifically addressing salaried positions available from established game companies as a career choice. It would be a completely separate article if I were to talk (enthusiastically) about the indie gaming scene and entrepreneurial business models in gaming.<br />
<br />
It's a super exciting time in the game industry with the advent of the mobile platforms. There are all kinds of incredible opportunities for young people to get into games and completely bypass the traditional established game publishers.<br />
<br />
And, you sure don't need a college education to pursue these opportunities either. However, that really isn't what my previous article was about. <br />
<br />
For a more colorful observation on this topic, I direct you to a <a href="http://cbloomrants.blogspot.com/2013/03/03-31-13-some-gdc-observations.html">blog post</a> by my friend Charles Bloom.<br />
<h4>
(6) This one nobody argued with me about. The game industry as a 'career' is extremely unstable, and this simply reinforces how important it is not to limit yourself to this very iffy field. Just to hammer home that point, <a href="http://www.shacknews.com/news">read these headlines</a> that just came out *today*! Note the reward for one team shipping a game is they all lose their jobs; hurray!</h4>
<div>
By the way, to make my point futher about the predatory practices of these 'gaming colleges', it's more than a little bit amusing that Google AdWords keeps inserting 'game design school' advertisements at the top of these posts now. It's probably not helping their cause much...</div>
<div>
<h2 style="background-color: white; border: 0px; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 22px; margin: 0.5em 0px 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://www.shacknews.com/article/78547/square-enix-la-office-again-hit-with-layoffs" style="border: 0px; color: #021e6f; cursor: pointer; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Square Enix LA office again hit with layoffs</a></h2>
<span class="byline" style="background-color: white; border: 0px; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 11px; margin: 0px 0px 0.2em; outline: 0px; padding: 0px; vertical-align: baseline;">by John Keefer, Apr 03, 2013 1:00pm PDT</span><span style="background-color: white; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 14px; line-height: 17px;"></span><br />
<div class="tags" style="background-color: white; border: 0px; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 12px; margin: 0.5em 0px; outline: 0px; padding: 0px 6px 0px 0px; vertical-align: baseline;">
<span style="border: 0px; font-family: inherit; font-style: italic; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Related Topics – </span><a href="http://www.shacknews.com/tag/square-enix" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Square Enix</a>, <a href="http://www.shacknews.com/tag/layoffs" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Layoffs</a></div>
<div class="summary" style="background-color: white; border: 0px; clear: both; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 14px; line-height: 17px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://www.shacknews.com/article/78547/square-enix-la-office-again-hit-with-layoffs" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><img alt="Square Enix LA office again hit with layoffs" class="little-promo drop-shadow-small" src="http://cf.shacknews.com/images/20130326/6852a1d6596d2b2e3e3492ac109edbe5_25138.nssp.jpg" style="-webkit-box-shadow: rgb(102, 102, 102) 2px 2px 2px; background-color: #d0d0d0; border: 0px; box-shadow: rgb(102, 102, 102) 2px 2px 2px; float: right; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px 5px 10px 10px; outline: 0px; padding: 2px; vertical-align: baseline;" /></a><br />
<div style="border: 0px; font-family: inherit; font-style: inherit; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div style="border: 0px; font-family: inherit; font-style: inherit; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
The Los Angeles offices of Square Enix America have <a href="http://www.shacknews.com/article/77079/square-enix-north-american-office-hit-with-layoffs" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">again</a>been hit with an unspecified number of layoffs, reportedly including the CEO and head of marketing. The office houses mostly marketing and public relations staff.</div>
<div style="border: 0px; font-family: inherit; font-style: inherit; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
<div style="border: 0px; font-family: inherit; font-style: inherit; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
<div style="border: 0px; font-family: inherit; font-style: inherit; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
<div style="border: 0px; font-family: inherit; font-style: inherit; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
<div style="border: 0px; font-family: inherit; font-style: inherit; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
</div>
<h2 style="border: 0px; font-size: 22px; margin: 0.5em 0px 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://www.shacknews.com/article/78545/activision-lays-off-40-at-deadpool-developer" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Activision lays off 40 at Deadpool developer</a></h2>
<span class="byline" style="border: 0px; font-size: 11px; margin: 0px 0px 0.2em; outline: 0px; padding: 0px; vertical-align: baseline;">by John Keefer, Apr 03, 2013 12:00pm PDT</span><br />
<div class="tags" style="border: 0px; font-size: 12px; margin: 0.5em 0px; outline: 0px; padding: 0px 6px 0px 0px; vertical-align: baseline;">
<span style="border: 0px; font-family: inherit; font-style: italic; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Related Topics – </span><a href="http://www.shacknews.com/game/deadpool" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Deadpool</a>, <a href="http://www.shacknews.com/tag/xbox-360" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">xbox 360</a>, <a href="http://www.shacknews.com/tag/playstation-3" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">PlayStation 3</a>, <a href="http://www.shacknews.com/tag/activision" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Activision</a>, <a href="http://www.shacknews.com/tag/pc" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">PC</a>, <a href="http://www.shacknews.com/tag/high-moon-studios" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">High Moon Studios</a></div>
<div class="summary" style="border: 0px; clear: both; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://www.shacknews.com/article/78545/activision-lays-off-40-at-deadpool-developer" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><img alt="Activision lays off 40 at Deadpool developer" class="little-promo drop-shadow-small" src="http://cf.shacknews.com/images/20120815/deadpool_gamescom_two-are-better-than-one_23104.nssp.jpg" style="-webkit-box-shadow: rgb(102, 102, 102) 2px 2px 2px; background-color: #d0d0d0; border: 0px; box-shadow: rgb(102, 102, 102) 2px 2px 2px; float: right; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px 5px 10px 10px; outline: 0px; padding: 2px; vertical-align: baseline;" /></a><br />
<div style="border: 0px; font-family: inherit; font-style: inherit; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div style="border: 0px; font-family: inherit; font-style: inherit; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://www.shacknews.com/game/deadpool" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Deadpool</a> is scheduled to come out sometime <a href="http://www.shacknews.com/article/77817/deadpool-targeting-summer-release" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">this summer</a>, and development is winding down at High Moon Studios. Unfortunately for 40 full-time members of the dev team, that not only meant an end to their work on the game, but a pink slip as well.</div>
</div>
<br />
<div style="border: 0px; font-family: inherit; font-style: inherit; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
</div>
</div>
<br />
<br />
<br />
<h2 style="background-color: white; border: 0px; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 22px; margin: 0.5em 0px 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://www.shacknews.com/article/78558/disney-closes-lucasarts-cancels-current-projects" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Disney closes LucasArts, cancels current projects</a></h2>
<span class="byline" style="background-color: white; border: 0px; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 11px; margin: 0px 0px 0.2em; outline: 0px; padding: 0px; vertical-align: baseline;">by Steve Watts, Apr 03, 2013 10:40am PDT</span><span style="background-color: white; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 14px; line-height: 17px;"></span><br />
<div class="tags" style="background-color: white; border: 0px; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 12px; margin: 0.5em 0px; outline: 0px; padding: 0px 6px 0px 0px; vertical-align: baseline;">
<span style="border: 0px; font-family: inherit; font-style: italic; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Related Topics – </span><a href="http://www.shacknews.com/tag/lucasarts" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">LucasArts</a>, <a href="http://www.shacknews.com/tag/star-wars-series" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Star Wars Series</a></div>
<div class="summary" style="background-color: white; border: 0px; clear: both; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 14px; line-height: 17px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://www.shacknews.com/article/78558/disney-closes-lucasarts-cancels-current-projects" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><img alt="Disney closes LucasArts, cancels current projects" class="little-promo drop-shadow-small" src="http://cf.shacknews.com/images/20120605/e3_14_22392.nssp.jpg" style="-webkit-box-shadow: rgb(102, 102, 102) 2px 2px 2px; background-color: #d0d0d0; border: 0px; box-shadow: rgb(102, 102, 102) 2px 2px 2px; float: right; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px 5px 10px 10px; outline: 0px; padding: 2px; vertical-align: baseline;" /></a><br />
<div style="border: 0px; font-family: inherit; font-style: inherit; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div style="border: 0px; font-family: inherit; font-style: inherit; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
Disney has shut down LucasArts and canceled all current projects, as part of a move toward licensing out the Star Wars brand to other publishers. This would impact <a href="http://www.shacknews.com/game/star-wars-1313" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Star Wars 1313</a>and <a href="http://www.shacknews.com/article/78162/star-wars-first-assault-is-a-downloadable-precursor-to-battlefront" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Star Wars: First Assault</a>.</div>
</div>
<br />
<br />John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com11tag:blogger.com,1999:blog-23430315.post-90929192803391820312013-04-01T15:47:00.001-07:002013-04-03T15:52:03.641-07:00So your teenager tells you they want to 'make video games' for a living...<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgE_Ubs9CmsRYXqJsbpmvvkWh1RSAFx6wXdwKaVLRK-fkCEGaqjLhuJ66B6yDsvx9YCJzKsZ6pP9PhdYbhzPgceHbmxNmzV7h_VFkaw9Ycs9Zfl1upYjnKlOOUx9mXXSQK7Rog3Nw/s1600/introduction-to-mmog-design.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgE_Ubs9CmsRYXqJsbpmvvkWh1RSAFx6wXdwKaVLRK-fkCEGaqjLhuJ66B6yDsvx9YCJzKsZ6pP9PhdYbhzPgceHbmxNmzV7h_VFkaw9Ycs9Zfl1upYjnKlOOUx9mXXSQK7Rog3Nw/s400/introduction-to-mmog-design.jpg" width="400" /></a></div>
<div>
<span style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;"><br /></span>
<span style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;"><br />If you are the kind of person who doesn't like to read a long article; then just watch this video. It says basically everything I am trying to say here but with one critical exception. This video tries to make a distinction between a 'good' game school and a bad one. My argument is if you have the talent to make it into the game industry, then you should not go to *any* game school. Attend a real college and get a proper education that will afford you the opportunity to land a job both in the game industry as well as a whole bunch of other fields as well. Do not limit yourself!!</span><br />
<span style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;"><br /></span>
<span style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">I have also written a <a href="http://codesuppository.blogspot.com/2013/04/some-clarifications-on-my-previous-post.html">follow up post </a>to this article to try to address some of the criticism it has received.</span>
<span style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;"><br /></span></div>
<iframe allowfullscreen="" frameborder="0" height="315" src="http://www.youtube.com/embed/zZczLMnWGAk" width="560"></iframe>
<br />
<div>
<span style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">I get asked advice over and over again throughout the years by friends and </span><span style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">acquaintances who have children that </span><span style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">desperately</span><span style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;"> want to know 'how to become a game developer' when they grow up.</span><span style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;"> Most of the time this young person thinks their qualifications come from the fact that they really, really, like to play video games a lot; and I mean like a *LOT*. When I hear this it always makes me think of the saying,<i><b> "Just because you like to eat candy doesn't mean you want to work in a candy factory....."</b></i></span><br />
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
I tend to keep giving the same explanations out time and time again and it's always awkward because I have to explain to people that this is almost always simply not a realistic expectation.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
What I decided to do today is to write up the explanation here on my blog so that, in the future, I can just refer people to it rather than having to get into a protracted exchange each time.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
I might add that this is becoming more and more of an issue because of the incredible amount of predatory behavior that is taking place in our culture. There are numerous institutes, colleges, and specialized programs all designed to convince young people that they can have a bright and promising future working in the video game industry. These programs take advantage of the millions of young people who have a deep passion and love for games by filling them with false hope of jobs and careers that simply do not exist, all the while aggressively running these young people and their families into enormous debt. Maybe your son or daughter came home with a bright shiny brochure about how in just a short 18 month program they too can be a video game designer and get a job at Electronic Arts. Don't believe it.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Let's get one thing clear right off of the bat. The chances of you getting a degree or certificate at a technical institute in 'computer game design' and getting an actual salaried position in the game industry is virtually non-existent. It's like the odds of winning the lottery or something.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Now, this article is not meant to dissuade young people from following their interest in computer games and even game design. That is not what I am trying to convey here at all. What I am trying to make clear is that going to a technical school or specialized program and spending thousands and thousands of dollars on the hopes of making a career in the video game industry is an extremely poor investment. </div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
If you are a young person interested in being involved in the creative process of video games then I would strongly encourage you to join clubs, practice making games on your own using a wide variety of free tools which are available, and getting involved in things like '<a href="http://globalgamejam.org/">The Global Game Jam</a>'.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
However, if you insist on making a career out of it, here is my practical advice on the matter.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
There are really only two good paying salaried positions in the game industry that are available. </div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
The first is as a software engineer and the second is as an artist.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
These are real jobs which can pay well and be very rewarding. </div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Every other job you may have heard about in the game industry does not offer a similar path. Yes, there are people in the game industry that have the job title of 'Game Designer' but few of them got that job because they went to college and got a degree in game design. Some probably have, but I would imagine that for every one person who has a salaried position as an honest to goodness game designer in the industry there are thousands and thousands of other people who can't even get their foot in the door.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Most people who have jobs as a 'game designer' got that job based on who else they already knew in the game industry and by leveraging existing personal relationships.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
So, my general advice to you, is that if you don't want to work on games as an artist or as a software engineer, then you probably should forget about making a 'career' out of it. You most certainly shouldn't pay thousands of dollars to some technical institute who promises an industry job when you are done. If you are a parent and your child is trying to talk you into paying for a game design school please don't fall for it. These are largely a scam which prey on the hopes and dreams of young people.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
And, well, come to think of it, in some ways the entire video game industry preys on the hopes and dreams of young people. That's why only a small percentage of people who started out working in the video game industry are still there 10 years later. At a presentation at GDC this year someone stated, and I'm not sure if these statistic is absolutely true but it's probably in the ballpark, is that the average career lifespan in film and games is 4 years and only 3% manage to make it to the ten year mark. It is for this reason alone that getting some kind of a 'specialized' degree is a really, really, bad idea. Most of those who do have a career in the industry experience a boom-bust-cycle which requires them to constantly change jobs and experience a tremendous amount of anxiety and stress. </div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
But, more on that later. </div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Personally, I would not recommend that anyone get a degree from any 'institute' dedicated to 'getting into the game industry'. Yes, I know there are good programs out there and I know there are success stories. However, the real issue is that you probably don't want to limit your career opportunities to just this one single industry. And, really, you don't have to. The skills needed to be an artist or software engineer in the game industry are the same exact skills that are desired by numerous other companies as well. So, why limit your options by going to some 'gaming college'?? Why not go to a real University and get a real University degree so that once you find out how truly awful the game industry sucks, you can get a job at a company which recognizes the concept of things like free evenings and weekends??</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
So, let's take a look at the requirements.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Does your son or daughter want to be an artist working on video games? Great. Is your son or daughter a simply mind-blowingly amazingly talented artist? Do they do art at a young age that looks like it could have been done by the best animators at Walt Disney or Pixar? </div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
If the answer to that question is 'no', then your child is not qualified to be in the game industry as an artist. You need to face this reality very early on.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
For example, check out the portfolios at <a href="http://www.gameartisans.org/">www.gameartisans.org</a> </div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
If your child cannot do art of this caliber, then they probably should not waste your money at art school.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Now, you might object, and say that "my son or daughter isn't that good yet because they haven't gone to college for four years to 'learn' all of that." Let me be clear. You do not go to college to 'learn' talent. You are either born with it or not. You go to college to learn technique and how to use complex and sophisticated computer software which is used by the graphics art industry, but college simply does not and cannot 'teach' talent.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Let me give you an example. A number of years ago I interviewed a young person to fill an entry level position as a graphics artist at a game company I worked for. This young person was only 19 years old and had never been to college for art.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Here is a link to her website. This is the quality of art that she presented me at the age of 19!</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<a href="http://lackadaisy.foxprints.com/exhibit.php?exhibitid=396">http://lackadaisy.foxprints.com/exhibit.php?exhibitid=396</a></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Now, Tracy is extremely talented, but it is this level of natural talent that the game industry seeks out.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Be realistic with yourself and your child. If they do not have this level of talent, please don't encourage them to waste their time and your money going to college to be a 'game artist'. They will never get a job and never get a degree that will be helpful to them.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Think about those kids that go onto 'American Idol' and somehow convince themselves that they are great singers when, in fact, they sound like an out-of-tune screeching owl. Someone should have told them that first. You are doing no favors to your child by encouraging them to pursue a field for which they lack natural talent.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Now, if your child does have that level of artistic talent, then things are actually pretty positive because, it turns out, that the same exact skill set needed to do video game artwork is also the same skill set needed to do special effects for film, television, web-design, and advertising. There are many, many, jobs available both inside and outside of the game industry if you have a high level of talent, a strong work ethic, and the technical skill set necessary to produce this kind of content.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
You might think that after stressing the level of talent required that I might argue that, maybe, a college education isn't even necessary. While it is true that you can make it in the industry on sheer talent alone, a college education really does offer a lot of value. Today the kind of art that people are creating involves a whole lot more than just a sheet of paper and a pencil. Extremely complex and sophisticated computer software is required to produce the kind of art used by video games and movies. And, while it is possible to learn this software on your own, a formal college education is extremely valuable.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Once again, I do not recommend institutes devoted exclusively to the video game industry. I just believe they are too narrow and limiting. One excellent school is the <a href="http://www.scad.edu/">Savannah College of Art and Design</a>. When researching a good art school make sure it is one which focuses on practical graphic art and design; one taught by industry professionals and one where most of the graduates immediately get jobs at established companies. Too many art schools focus on fine art. Now, if that is what you want to pursue ok, but don't expect to get a job based on that background.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Now that I have covered the topic of 'game artist', next let's focus on 'game programmer'.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Many young people will readily admit that they don't have the artistic talent necessary to be a game artist but are confident that they have what it takes to be a game programmer.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
My first question to that statement always begins (and usually ends) with 'How good is your kid at math?' If the answer is anything other than 'My kid is great at math and we go to state math competitions on a regular basis' then, really, forget about it.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
There are lots of jobs in computer software where you do not, necessarily, have to be really, really, really, good at math. Computer games are not one of them. </div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
When I went to college 30 years ago, a degree in computer science required three semesters of calculus, differential equations, linear algebra, and matrix algebra, and things have only become more demanding since then.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
The amount of mathematics behind the technology in computer video games today is staggering. Even if you don't use that level of math on a daily basis you certainly need the background and to have the kind of mind that is capable of grasping and learning it. </div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
If your kid is not an expert at math, don't even bother discussing the subject of being a computer game programmer. It's a non-starter.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
The degree you need to pursue in college to become a computer game programmer is 'computer science'. Good computer science programs are available at M.I.T., Stanford, and the University of Illinois for example. Yes, <b>those </b>schools, not some 'computer gaming technical' institute. And, yes, there are many great state schools with excellent computer science and engineering programs. These are all great places to to get the kind of degree that will give you a strong grounding in the field of computer science and software engineering.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
The good news here, is that if your child can score a 32 on their ACT and get into one of these colleges and walk away with a degree in computer science, not only are they qualified to get a job in the video game industry, they are qualified to get a job at Google, Microsoft, Apple, Boeing, General Dynamics, Lockheed, etc. etc.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
If you somehow had the idea in your head that working on computer video games was 'easier' than working in these other fields, well, you must dissuade yourself of that notion right away. Some of the most advanced and cutting edge work in the fields of computer graphics, artificial intelligence, and computer simulation, happens in the video game industry. We have major scientific conferences where our work is presented. And, yes, there is a lot of math involved.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Let me give you a case in point. Here are some papers recently published by my colleague Mathias Muller. All of the work presented in these papers we are applying directly to computer games currently in development.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<a href="http://www.matthiasmueller.info/publications/FTLHairFur.pdf">http://www.matthiasmueller.info/publications/FTLHairFur.pdf</a></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<a href="http://www.matthiasmueller.info/publications/masscon_sca.pdf">http://www.matthiasmueller.info/publications/masscon_sca.pdf</a></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<a href="http://www.matthiasmueller.info/publications/sca2012cloth.pdf">http://www.matthiasmueller.info/publications/sca2012cloth.pdf</a></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
These three papers, respectively, discuss techniques for how to simulate hair, fluids, and clothing suitable for use in computer video games.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Another point that needs to be raised, while you can have a degree in computer science from M.I.T. or Stanford, and Microsoft, Apple, or Google, might give you a job based on your education alone, the game industry will not. Boeing will hire you even if you have never personally built a fighter jet in your spare time. In contrast, it is extremely rare for game company to hire a software engineer who has not worked on a game before.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Really the only way to get a job in the game industry is based on your previous work on games. This may seem like a bit of a catch-22, and it kind of is, but that's just how things work.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
The video game industry is project based, it closely mirrors the movie industry in that way. No video game company can afford to hire someone who can 'learn' on the job. There are sometimes very limited internship opportunities in the game industry but, really, for an internship to be effective some senior people need to act as a mentor, which just takes them away from their other duties. Game companies care about one thing and one thing only, and that is shipping their game on-time. This singular focus is all consuming and leaves little room for internship and on the job training.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
You might wonder, how does anyone get a job in the game industry if the only way to get a job in the game industry is to have previously worked on a game?</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
The answer to this is simple, you have to work on games 'for fun' as personal projects on your own time and without pay. Fortunately with the advent of the Iphone and Android platforms this is today a completely realistic goal.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
An individual or small team can write a game for a mobile device simply as a learning experience. If their game demonstrates a high degree of technical skill and professionalism, that is exactly the right ingredients necessary to land an entry level position in the game industry.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Virtually every single software engineer in the game industry is, to a large extent, self-taught. Sure, most have gone to college and have degrees, but they were virtually all programming computers since a young age and much of the work they have done was self-directed and driven.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
This is pretty much universally true.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
If your son or daughter is not already programming the computer since an early age, teaching themselves programming and actually doing it, then almost for sure they don't have what it takes to make it in the game industry.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
I know of almost no successful software engineer in the video game industry who did not teach themselves how to program at a young age and write their own computer games before they got their first job. </div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
So, if your son or daughter's idea of 'fun' is reading a book on C++ programming, neural networks, graphs and data structures, algorithms, and other such exciting material, then maybe they do have what it takes to work in the game industry. However, if they are not already teaching themselves, I don't think any 'game institute' is going to make any big difference. </div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;">Years ago it was common for people working as computer game programmers to have no college degree at all but, over time, the process of software development has really evolved and changed. Years ago an entire computer game was written by a single guy. Today, a massive team of people work on a computer game in code bases comprising millions of lines of code. A completely 'self-taught' computer programmer does not have the training necessary to work with a large team. They have not learned the proper design principles and methodologies (such as Agile) or have the necessary background in mathematics previously discussed. These skills and the maturity which comes with a well rounded education go a long way towards getting a person prepared to work in a professional team environment. </span></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
So, to recap:</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<h4>
<br />
<br />
<ul>
<li><span style="text-align: -webkit-auto;">There are only really two high demand and high salary career opportunities in the game industry; artist or software engineer.</span></li>
</ul>
<ul>
<li><span style="text-align: -webkit-auto;">To be an artist you must have a high level of talent and ideally attend a four year University with an excellent graphics art program which teaches technique and how to use advanced and sophisticated software used in the industry.</span></li>
</ul>
<ul>
<li><span style="text-align: -webkit-auto;">To be a game programmer, you must be extremely good at math, self-directed and driven; programming games 'for fun' even at an early age, and pursue a four year degree in computer science at an excellent school.</span></li>
</ul>
<br />
</h4>
</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Finally, I will address what is probably the most important issue of all. Should you even want to work in the game industry at all?</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
This is a much more serious question than you might think. If you like to see your wife and children, then the game industry might not be for you. If you like to go home in the evenings and have weekends, then the game industry might not be for you. If you would like a job where you do not, on a daily basis, live in fear of having your project canceled or being laid off, then the game industry may not be fore you. If you don't want to work at a company that is driven by often insane and irrational schedules with completely unrealistic expectations about what can be done in a given period of time, then the game industry might not be for you.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
The video game industry is project driven, just like the movie industry. <span style="text-align: -webkit-auto;">In other fields when you work hard on a project and complete it, you might be rewarded with a bonus. In the video game industry, your reward for working hard on a project, often times, is to be laid off. </span><span style="text-align: -webkit-auto;">Game companies continue throwing bodies at a project to try to get it out the door. Once it does ship, they now have way too many people on board and the cycle is often a massive layoff.</span></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;">Most video game projects that get started end up being canceled. There are many people in the industry who have worked for years and years, but never actually had a shipped game. Every project they have ever worked on, and sacrificed their time and talent on, ended up getting canceled. This can be really devastating emotionally to deal with.</span></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
The video game industry knows just how much young people have a romantic notion about 'working on games'. They take advantage of that. When you tour a video game company you might be impressed by the free snacks, the free meals, the ping-pong table, and the pinball machines. All of that must look fun and seductive but the real reason all of that stuff exists in the office, is because the game company executives don't want you to go home. (<i>Think of it like Hansel and Gretel finding a gingerbread house, it may look delicious but something dangerous lurks inside and you will never leave....</i>) Game company executives, they don't want you to have a life outside of the office; your apartment is nothing more than a place to sleep and, don't worry, those comfy looking couches in the office have a dual purpose..</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
Now, for a young person, 22 years old, this isn't necessarily a big deal. They love the work, and it is fun, and what else do they really have going on in their life anyway?</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
But, as you get older, and try to make a 'career' out of this industry, it simply doesn't mesh with a quality way of life. For about ten years of my career my children virtually never saw me. I was a completely absent father. I deeply regret that today.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
If you search the Internet for<a href="http://ea-spouse.livejournal.com/274.html"> articles on this topic</a> you will find that the video game industry is charred with the wreckage of families torn apart by these practices.</div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
I would like to add yet one more note of caution about these 'gaming institutes' and 'technical schools'. Let's say you make it through one of those programs and actually manage to get a job in the game industry. Realize that the game company is going to know that your options are limited. Your salary will reflect that. If you had gotten a full four year degree in computer science from a really good University, you could command an impressive salary with decent working environment and benefits from any one of a number of technology companies. </div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;">I know this article may get some blowback from these colleges and institutes who take so much pride in their game courses. I am merely expressing my personal opinion here. I have been working in the game industry continuously for literally over 30 years. I wrote my first games on an Apple-II computer for Milliken Publishing Company back in 1982.</span></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;"><br /></span></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;">I have worked with many of the top game companies over the years and I have been involved in numerous game projects. I am still in the industry today though I no longer work directly for a game company; for all of the reasons cited above.</span></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;"><br /></span></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;">My argument here is simple. If you have the talent and drive necessary to work on the most advanced computer games and technology then you have the talent and drive necessary to get a real university degree from a college with a great reputation. I cannot see any reason, in any way shape or form, that you should limit yourself or your career opportunities by pursing a limited degree program dedicated solely to the game industry.</span></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;"><br /></span></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;">The simple fact of the matter is that there are thousands more young people who dream of some day becoming a 'video game designer' for a big game company than their are actual paid salaried positions available. </span></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;"><br /></span></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;">Sometimes I use this analogy. To say that you want to be a 'game designer' is somewhat akin to saying you want to 'be a movie star'. You probably have about as much chance.</span></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;"><br /></span></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;">My advice is to the parents. Unless your child has the talent and skill set I have described above as either an artist or a software engineer, then I would strongly, strongly, strongly, recommend that you do not waste your money or your child's life by letting them pursue a career 'opportunity' as a game designer.</span></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;"><br /></span></div>
<div style="orphans: 2; text-align: -webkit-auto; widows: 2;">
<span style="font-family: Tahoma;">They have a better chance of developing a best selling Iphone or Android game on their own personal time than they do getting a job at Electronic Arts because they went to a 'game design' program at some technical institute. They probably have a better chance of winning the lottery quite frankly.</span></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
<br /></div>
<div style="font-family: Tahoma; orphans: 2; text-align: -webkit-auto; widows: 2;">
------------------------------------<br />
<br />
<h2>
A little postscript here. </h2>
<br />
One of the main points of this article was that the video game industry is unstable and not a good place to try to make a career. I say repeatedly that you should not limit your career opportunities to this industry alone.<br />
<br />
And, ironically, what happens within 24 hours of my posting this and it going viral?<br />
<br />
Major layoffs at game companies and Walt Disney completely shuts down one of the most famous game studios of all time 'Lucas Arts'.<br />
<br />
I don't know how much more clearly I could make my point than this. Every one of these people, their lives are now completely disrupted. They will have to move, try to find new jobs, be put at financial risk and go through incredible stress and uncertainty. If they are married and have children this will be especially difficult to deal with.<br />
<br />
It's not that you don't get laid off in other industries but in the game industry it's practically 'expected' on a frequent basis. And, more to the point of this article, many of these people will face challenges trying to migrate to other fields than computer games alone if they can't find any openings there. <br />
<br />
<br />
<div style="background-color: #f7f0e9; color: #632035; font-family: helvetica, arial, verdana, 'trebuchet ms', sans-serif; font-size: 13px; line-height: 18px;">
<h2 style="background-color: white; border: 0px; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 22px; margin: 0.5em 0px 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://www.shacknews.com/article/78547/square-enix-la-office-again-hit-with-layoffs" style="border: 0px; color: #021e6f; cursor: pointer; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Square Enix LA office again hit with layoffs</a></h2>
<span class="byline" style="background-color: white; border: 0px; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 11px; margin: 0px 0px 0.2em; outline: 0px; padding: 0px; vertical-align: baseline;">by John Keefer, Apr 03, 2013 1:00pm PDT</span><span style="background-color: white; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 14px; line-height: 17px;"></span><br />
<div class="tags" style="background-color: white; border: 0px; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 12px; margin: 0.5em 0px; outline: 0px; padding: 0px 6px 0px 0px; vertical-align: baseline;">
<span style="border: 0px; font-family: inherit; font-style: italic; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Related Topics – </span><a href="http://www.shacknews.com/tag/square-enix" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Square Enix</a>, <a href="http://www.shacknews.com/tag/layoffs" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Layoffs</a></div>
<div class="summary" style="background-color: white; border: 0px; clear: both; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 14px; line-height: 17px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://www.shacknews.com/article/78547/square-enix-la-office-again-hit-with-layoffs" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><img alt="Square Enix LA office again hit with layoffs" class="little-promo drop-shadow-small" src="http://cf.shacknews.com/images/20130326/6852a1d6596d2b2e3e3492ac109edbe5_25138.nssp.jpg" style="-webkit-box-shadow: rgb(102, 102, 102) 2px 2px 2px; background-color: #d0d0d0; border: 0px; box-shadow: rgb(102, 102, 102) 2px 2px 2px; float: right; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px 5px 10px 10px; outline: 0px; padding: 2px; vertical-align: baseline;" /></a><br />
<div style="border: 0px; font-family: inherit; font-size: 13px; font-style: inherit; line-height: 18px; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div style="border: 0px; font-family: inherit; font-size: 13px; font-style: inherit; line-height: 18px; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
The Los Angeles offices of Square Enix America have <a href="http://www.shacknews.com/article/77079/square-enix-north-american-office-hit-with-layoffs" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">again</a>been hit with an unspecified number of layoffs, reportedly including the CEO and head of marketing. The office houses mostly marketing and public relations staff.</div>
<div style="border: 0px; font-family: inherit; font-size: 13px; font-style: inherit; line-height: 18px; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
<div style="border: 0px; font-family: inherit; font-size: 13px; font-style: inherit; line-height: 18px; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
<div style="border: 0px; font-family: inherit; font-size: 13px; font-style: inherit; line-height: 18px; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
<div style="border: 0px; font-family: inherit; font-size: 13px; font-style: inherit; line-height: 18px; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
<div style="border: 0px; font-family: inherit; font-size: 13px; font-style: inherit; line-height: 18px; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
</div>
<h2 style="border: 0px; font-size: 22px; margin: 0.5em 0px 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://www.shacknews.com/article/78545/activision-lays-off-40-at-deadpool-developer" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Activision lays off 40 at Deadpool developer</a></h2>
<span class="byline" style="border: 0px; font-size: 11px; margin: 0px 0px 0.2em; outline: 0px; padding: 0px; vertical-align: baseline;">by John Keefer, Apr 03, 2013 12:00pm PDT</span><br />
<div class="tags" style="border: 0px; font-size: 12px; line-height: 18px; margin: 0.5em 0px; outline: 0px; padding: 0px 6px 0px 0px; vertical-align: baseline;">
<span style="border: 0px; font-family: inherit; font-style: italic; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Related Topics – </span><a href="http://www.shacknews.com/game/deadpool" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Deadpool</a>, <a href="http://www.shacknews.com/tag/xbox-360" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">xbox 360</a>, <a href="http://www.shacknews.com/tag/playstation-3" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">PlayStation 3</a>, <a href="http://www.shacknews.com/tag/activision" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Activision</a>, <a href="http://www.shacknews.com/tag/pc" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">PC</a>, <a href="http://www.shacknews.com/tag/high-moon-studios" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">High Moon Studios</a></div>
<div class="summary" style="border: 0px; clear: both; font-size: 13px; line-height: 18px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://www.shacknews.com/article/78545/activision-lays-off-40-at-deadpool-developer" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><img alt="Activision lays off 40 at Deadpool developer" class="little-promo drop-shadow-small" src="http://cf.shacknews.com/images/20120815/deadpool_gamescom_two-are-better-than-one_23104.nssp.jpg" style="-webkit-box-shadow: rgb(102, 102, 102) 2px 2px 2px; background-color: #d0d0d0; border: 0px; box-shadow: rgb(102, 102, 102) 2px 2px 2px; float: right; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px 5px 10px 10px; outline: 0px; padding: 2px; vertical-align: baseline;" /></a><br />
<div style="border: 0px; font-family: inherit; font-style: inherit; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div style="border: 0px; font-family: inherit; font-style: inherit; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://www.shacknews.com/game/deadpool" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Deadpool</a> is scheduled to come out sometime <a href="http://www.shacknews.com/article/77817/deadpool-targeting-summer-release" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">this summer</a>, and development is winding down at High Moon Studios. Unfortunately for 40 full-time members of the dev team, that not only meant an end to their work on the game, but a pink slip as well.</div>
</div>
<br />
<div style="border: 0px; font-family: inherit; font-size: 13px; font-style: inherit; line-height: 18px; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<br /></div>
</div>
</div>
<br style="background-color: #f7f0e9; color: #632035; font-family: helvetica, arial, verdana, 'trebuchet ms', sans-serif; font-size: 13px; line-height: 18px;" />
<br style="background-color: #f7f0e9; color: #632035; font-family: helvetica, arial, verdana, 'trebuchet ms', sans-serif; font-size: 13px; line-height: 18px;" />
<br style="background-color: #f7f0e9; color: #632035; font-family: helvetica, arial, verdana, 'trebuchet ms', sans-serif; font-size: 13px; line-height: 18px;" />
<h2 style="background-color: white; border: 0px; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 22px; line-height: 18px; margin: 0.5em 0px 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://www.shacknews.com/article/78558/disney-closes-lucasarts-cancels-current-projects" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Disney closes LucasArts, cancels current projects</a></h2>
<span class="byline" style="background-color: white; border: 0px; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 11px; line-height: 18px; margin: 0px 0px 0.2em; outline: 0px; padding: 0px; vertical-align: baseline;">by Steve Watts, Apr 03, 2013 10:40am PDT</span><span style="background-color: white; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 14px; line-height: 17px;"></span><br />
<div class="tags" style="background-color: white; border: 0px; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 12px; line-height: 18px; margin: 0.5em 0px; outline: 0px; padding: 0px 6px 0px 0px; vertical-align: baseline;">
<span style="border: 0px; font-family: inherit; font-style: italic; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">Related Topics – </span><a href="http://www.shacknews.com/tag/lucasarts" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">LucasArts</a>, <a href="http://www.shacknews.com/tag/star-wars-series" style="border: 0px; color: black; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Star Wars Series</a></div>
<div class="summary" style="background-color: white; border: 0px; clear: both; color: #333333; font-family: 'Helvetica Neue', Helvetica, Arial, clean, sans-serif; font-size: 14px; line-height: 17px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">
<a href="http://www.shacknews.com/article/78558/disney-closes-lucasarts-cancels-current-projects" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;"><img alt="Disney closes LucasArts, cancels current projects" class="little-promo drop-shadow-small" src="http://cf.shacknews.com/images/20120605/e3_14_22392.nssp.jpg" style="-webkit-box-shadow: rgb(102, 102, 102) 2px 2px 2px; background-color: #d0d0d0; border: 0px; box-shadow: rgb(102, 102, 102) 2px 2px 2px; float: right; font-family: inherit; font-style: inherit; font-weight: inherit; margin: 0px 5px 10px 10px; outline: 0px; padding: 2px; vertical-align: baseline;" /></a><br />
<div style="border: 0px; font-family: inherit; font-size: 13px; font-style: inherit; line-height: 18px; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
</div>
<div style="border: 0px; font-family: inherit; font-size: 13px; font-style: inherit; line-height: 18px; margin-bottom: 1em; margin-top: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
Disney has shut down LucasArts and canceled all current projects, as part of a move toward licensing out the Star Wars brand to other publishers. This would impact <a href="http://www.shacknews.com/game/star-wars-1313" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Star Wars 1313</a>and <a href="http://www.shacknews.com/article/78162/star-wars-first-assault-is-a-downloadable-precursor-to-battlefront" style="border: 0px; color: #021e6f; font-family: inherit; font-style: inherit; font-weight: bold; margin: 0px; outline: 0px; padding: 0px; text-decoration: none; vertical-align: baseline;">Star Wars: First Assault</a>.</div>
</div>
</div>
</div>
John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com92tag:blogger.com,1999:blog-23430315.post-31724268524953136142013-03-15T14:14:00.001-07:002013-03-15T14:14:06.965-07:00Open Source Binary EZ-Mesh File Reader<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_2tlTcI0cdcw4PQAA0OFKT5Gcbq-lw4hlx87haZ9yrsSpXyylRrfPYP7-fCGXYlGjDiAMhNVFh8KhfxvIUWI-ARc_0diCti_1M4-XeDjTX3kg7EpPXM171U_FFUj-67W_GghlXw/s1600/ezb.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="321" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_2tlTcI0cdcw4PQAA0OFKT5Gcbq-lw4hlx87haZ9yrsSpXyylRrfPYP7-fCGXYlGjDiAMhNVFh8KhfxvIUWI-ARc_0diCti_1M4-XeDjTX3kg7EpPXM171U_FFUj-67W_GghlXw/s400/ezb.png" width="400" /></a></div>
<br />
The Binary EZ-Mesh file reader project, tools, and documentation can be found at the following <a href="https://code.google.com/p/ezb/">Google Code project</a>.<br />
<br />
Quite some time ago I created a graphics file format that I call 'EZ-Mesh'. My design goal was to create an extremely 'easy' way to represent 3D graphics data without requiring a massive amount of source code or bloated API.<br />
<br />
Originally the file representation for this data was always in XML but, as we all know, XML can get pretty bloated and take some time to parse. We are currently using EZ-Mesh in a bunch of our APEX SDK samples at NVIDIA; specifically all of our clothing samples. We have some very high poly count samples for clothing in the APEX SDK and needed to load them on consoles and tablets. Parsing massive XML files did not seem like the way to go; so thus was born the binary EZ-mesh format.<br />
<br />
Below is a short video sample of a character being mocked up with clothing for the APEX SDK<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="http://www.youtube.com/embed/_VCJbIf8jzI" width="560"></iframe>
<br />
<br />
The artists work in 3D Studio Max and Maya, but the goal for the programmer is to get the relevant data out of there and into our samples.<br />
<br />
Now, Autodesk provides a file format and SDK to do this. It's called the 'FBX' file format. However, to access the data in an .FBX file requires using a very large SDK and API. This API is really huge and complex and, of course, isn't likely to run in real-time on a console or tablet device.<br />
<br />
So, our need was a way to extract the relevant data from an .FBX file (meshes, materials, skeletons, and animation data) and convert it into something very simple that we could load directly into our samples.<br />
<br />
That's where EZ-Mesh comes in and the project I'm presenting today.<br />
<br />
When you sync to this project you will get a pre-built tool called 'MeshConvert', and a D3D9 viewer application called 'CodeSuppository'; as well as full source to a very tiny project that can read the contents of a binary EZ-Mesh into your own application.<br />
<br />
This project has an extensive ReadMe.txt file as well as source code documentation, so I won't go into all of the details here.<br />
<br />
The quick summary is that with the provided tool, you can convert a wide variety of graphics formats, including FBX, Ogre3d, UE3 PSK, and some others, into EZ-Mesh either XML or binary format.<br />
<br />
The conversion tool also has some command line switches that allow you to scale or rotate the input mesh before it is saved back out again. The conversion tool can also export several different graphics file formats in addition to EZ-Mesh.<br />
<br />
You can use the 'CodeSuppository' viewer app to preview the files.<br />
<br />
The 'EZ-Mesh' format is defined in a single header file 'EZMesh.h' which is a completely flat set of data structures contains mesh data, vertex buffers, index buffers, skeletons, and animation data.<br />
<br />
<h4>
<b>There is *no* API whatsoever!! </b>It's just a straight flat data structure that you can easily iterate through.</h4>
<br />
It is important to note that the EZ-Mesh data layout is *not* meant to be a production format. It's sole purpose is to provide an extremely easy to access flat layout for raw mesh, skeleton, and animation data. The purpose was not to be memory efficient.<br />
<br />
If you are writing a small sample and just want to quickly load a 3D graphics file with skeletons and animation data, then EZ-Mesh is for you. You can also use it as an intermediate format as part of a tool chain to ultimately convert the data into some final project ready format of your own choosing.<br />
<br />
The EZ-Mesh source code uses the PhysX/APEX SDK 'foundation' header files. These header files just define very simple and basic platform agnostic data types; such as defining floats and integers of various bit sizes and a basic set of vector classes.<br />
<br />
I do not have permission to redistribute the PhysX SDK foundation header files so what I have done with this open source project is created very simple proxy placeholders.<br />
<br />
It is recommended that you either get the actual header files from the PhysX SDK or do a global find replace on the source to remap the data types to whatever your project prefers to use.<br />
<br />
I have *not* included full source code to the mesh converter tool, import/export DLL plugins, or the viewer app. That was out of scope for this article today.<br />
<br />
You only need three source files (not counting the foundation headers) to load a binary EZ-Mesh into your application in a format ready for you to process, render, animate, etc.<br />
<br />
The first is: '<b><i>EZmesh.h'</i></b> which simply defines the flat data structures containing the mesh data. As stated before, it has no API and no dependencies other than the foundation primitive types.<br />
<br />
Next is:<b><u> 'MeshImportEZB.h'</u></b> which simply declares the API to parse a binary EZ-mesh blob into the data structure.<br />
<br />
Finally is: <u><b>'MeshImportEZB.cpp'</b></u> which contains the implementation to parse the binary EZ-mesh file, and to release temporary memory allocated when you are finished with it. This source has no dependencies on the STL, Windows, or anything else. In theory is should easily compile on almost any platform available; and it takes endian specific issues into account. In fact, the binary EZ-mesh format defaults to big-endian format just so that it will load a little bit faster on some console platforms.<br />
<br />
'MeshImportEZB.cpp' is just a little over 700 lines of code for the whole thing. The source comes with two *exporters* simply so you can step through the code and see how the various data structures are accessed. It also acts as a validation test for the importer.<br />
<br />
My design goal was to provide a tool chain that would produce a binary file containing all relevant 3D graphics data that could be loaded quickly and easily into a sample project.<br />
<br />
I think I have accomplished that goal, let me know if you find any of this useful.<br />
<br />John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com2tag:blogger.com,1999:blog-23430315.post-62977988681790916782013-03-13T14:46:00.001-07:002013-03-13T14:46:25.745-07:00My NVIDIA Developer Zone article<br />
<iframe width="420" height="315" src="http://www.youtube.com/embed/JyLX_t28zYc" frameborder="0" allowfullscreen></iframe>
<br />
I just had an article I wrote about optimization I did getting GPU PhysX particles integrated into Planetside 2 go live. You can read it at <a href="https://developer.nvidia.com/content/maximizing-physx-based-particle-effects-planetside-2">this link.</a>John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com2tag:blogger.com,1999:blog-23430315.post-89847489702786303332013-02-10T08:16:00.002-08:002013-02-10T08:31:53.694-08:00Backdoor : A simple code snippet to create a command console to a running application<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0-pgemy0MmyiKIGJNXYYis4J95LbeRSk1SvOTCUKcQyjbCsGXMlNwpT9GDYZXrvJCyPOnhQ6_LmRSIh2M-EMSMGxYQ6deHYHXf8ABFkv0OTg2UJzVd0MaNtEdZ7YygPhWGIMm0A/s1600/backdoor.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="136" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0-pgemy0MmyiKIGJNXYYis4J95LbeRSk1SvOTCUKcQyjbCsGXMlNwpT9GDYZXrvJCyPOnhQ6_LmRSIh2M-EMSMGxYQ6deHYHXf8ABFkv0OTg2UJzVd0MaNtEdZ7YygPhWGIMm0A/s400/backdoor.jpg" width="400" /></a></div>
<div style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 1.25em; max-width: 64em;">
<br /></div>
<div style="background-color: white; font-family: arial, sans-serif; font-size: 13px; line-height: 1.25em; max-width: 64em;">
<br /></div>
<div style="background-color: white; line-height: 1.25em; max-width: 64em;">
<span style="font-family: Arial, Helvetica, sans-serif;">Today I am presenting a '<a href="http://code.google.com/p/backdoor/">code snippet</a>' that I have found extremely useful. Here's the example use case. You are working on some application, it's a game, or whatever, and you need a command console but the application you are working on doesn't have one, or you aren't operating in a section of the code where you can easily just add new commands to the console interface. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />What you want to be able to do is send commands to your application, perhaps to toggle certain debug features on or off, or to receive warning and/or debug messages from it. I find this to be a fairly common thing I would like to be able to do.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br /><span style="font-size: small;"><span style="line-height: 1.25em;">What this code snippet does is that is creates a tiny buffer of shared memory to easily send and receive messages between two processes. It provides a little console application that let's you enter commands and </span><span style="line-height: 16.25px;">receive</span><span style="line-height: 1.25em;"> messages from the other application. It will also log all incoming messages to a text file that you can review after a session is complete.</span></span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-size: small;"><span style="line-height: 1.25em;"><br /></span></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-size: small;"><span style="line-height: 1.25em;">To review the header file defining the interface to the Backdoor class use this link:</span></span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-size: small;"><span style="line-height: 1.25em;"><br /></span></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-size: small;"><span style="line-height: 1.25em;"><a href="http://code.google.com/p/backdoor/source/browse/trunk/Backdoor.h">Backdoor.h</a></span></span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-size: small;"><span style="line-height: 1.25em;"><br /></span></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-size: small;"><span style="line-height: 1.25em;">To review the implementation of the code snippet; which is quite modest in size and reasonably well documented, use this link:</span></span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-size: small;"><span style="line-height: 1.25em;"><br /></span></span></span>
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-size: small;"><span style="line-height: 1.25em;"><a href="http://code.google.com/p/backdoor/source/browse/trunk/Backdoor.cpp">Backdoor.cpp</a></span></span></span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><span style="font-size: small;"><span style="line-height: 1.25em;"><br /></span></span>This project contains a code snippet 'Backdoor.cpp' which provides support for inter-process communication to easily send 'commands' between two processes on the same machine. The code snippet uses shared file memory buffers to very simply send a command stream between the two applications. It only supports a single message at a time; as it was not in the design scope for this snippet to handle a more complex multi-packet message stream.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;">The code was written for the Windows API but could fairly easily be modified for other platforms; just change the calls for memory mapped IO.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />This code snippet was designed for the sake of simplicity. It's design goal is to provide a super quick and dirty way to send messages using a chat client to a running application; a kind 'backdoor' way to implement a command console.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />As a convenience the class allows you to send a message using the 'printf' style format and when it receives messages it parses the input into an argc/argv format so that the application can easily digest the command and arguments.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif;"><br />You can download the source code and the sample application from this <a href="http://code.google.com/p/backdoor/">Google Code Project.</a></span><br />
<h2>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: small; font-weight: normal;">To test the app, run "Backdoor server" and then run 'Backdoor client' and you can send chat messages back and forth. The code is written in C++ and provided with a Visual Studio 2008 project file. It uses the Windows API for shared file memory access, but you could easily refactor it for other platforms.</span></h2>
</div>
John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com3tag:blogger.com,1999:blog-23430315.post-34881590572770755502012-11-12T15:40:00.002-08:002012-11-12T16:21:50.794-08:00Life Logging<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRAA6Y5apsykUsk0VtN1hv6X9DEZBOlVurXeSiEPlMdYDenz032jo34FJcbBNMZ1FCtnTu3uSygQaxVwRQJtHvxnMAXFiMba0mXvzaX6emdLcc_VGVVlwOJam7HLj9UkZjvSi5bg/s1600/memoto.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRAA6Y5apsykUsk0VtN1hv6X9DEZBOlVurXeSiEPlMdYDenz032jo34FJcbBNMZ1FCtnTu3uSygQaxVwRQJtHvxnMAXFiMba0mXvzaX6emdLcc_VGVVlwOJam7HLj9UkZjvSi5bg/s320/memoto.png" width="319" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><a href="http://memoto.com/">The Memoto Lifelogging Camera</a></td></tr>
</tbody></table>
I've been generally interested in the concept of '<a href="http://en.wikipedia.org/wiki/Lifelog">Lifelogging</a>' for quite some time. My main reason for this interest is because of what an absolutely terrible memory that I have. I cannot remember so much of my life it's really quite amazing. I have some general vague memories of things, and since I have started blogging/facebooking over past years, it has helped a bit I think. But, I have virtually no recollection of years and years of my life. Not only could I not tell you the names of any of my classmates I went to school with as a child, I couldn't tell you the name of the teacher, the name of the school, and in many cases I couldn't even remember where I lived. At the most I have just the tiniest snippets of memories for entire years of my life.<br />
<br />
The idea of 'Lifelogging' is that you use technology to act as an aid to memory. True 'lifelogging' would record audio, video, GPS, and biometrics of every second of your life. That is both extreme and ridiculous. There are other forms of 'Lifelogging' that we all experience today in the form of 'check-ins' and Facebook status updates.<br />
<br />
The new Facebook Timeline feature which, strangely so many people claim to hate, serves as a form of life-logging; by letting you scroll through your online history throughout time. Unlike others, I'm a big fan of this feature; that is up until Facebook decides to remove it or screw it up.<br />
<br />
The missing piece of technology to get 'Lifelogging' off the ground and out of the researcher stage has been a wearable device which is fairly innocuous. The first real attempt at this comes in the form of '<a href="http://memoto.com/">Memoto</a>'; the Lifelogging camera. This tiny wearable device snaps photographs, about two per minute, every moment of your waking life while you wear it. It also records GPS data. This data gets uploaded into a cloud service that is non-sponsored. Lifeloggers will pay an annual fee to Memoto for the privilege of storing their data online; the company guarantees that the data is yours; you own it, it is private, and encrypted. They won't sell your data to anyone else or allow it to be sponsored. <br />
<br />
Of course, there will be questions. Can the data be subject to subpoena? What about privacy issues? Are people going to be annoyed or object to you snapping pictures everywhere you go; especially in situations where they might presume it would be in private?<br />
<br />
I think they will, though it raises the question that most of us are being recorded already every time we are out in public by a myriad of surveillance cameras all the time; in the store, on the street, everywhere. We tune these out, but that's not quite the same as your picture being snapped twice a minute by your best friend sitting across the dinner table.<br />
<br />
There are already plenty of laws in place about having your conversation recorded without your permission; and that's why the 'Memoto' Lifelogging camera certainly will not be recording any audio. However, in our society at large, there is not currently any such prohibition against being photographed. We let that ship sale a long time ago.<br />
<br />
Personally, I tend not to get all worked up about the new age of personal mobile technology which surrounds us. In the first place, it's a big part of the business I'm in, so I'm hardly going to be resistant to it. So, as far as I'm concerned, bring on the wearable computers; I think we will sort it all out in the end. <br />
<br />
Now, back to the original goal of mine, that of using 'Lifelogging' technology to act as a form of photographic memory. I'm afraid there is a lot more to this problem domain than simply recording tens of thousands of photographs and uploading them to a hard drive. As anyone knows who has taken a digital camera on vacation, many of us can't even find the time to review all the pictures we took when we were doing something special! <br />
<br />
What the 'Memoto' team is offering is an organizational approach that gathers the data into 'Moments'. I like that idea, and I'm sure it will work fine if you want to go back to a previous day and do a review.<br />
<br />
However, I don't see how that will solve the problem of augmenting memory. Quickly day after day will add up, and you are not going to be able to review these 'moments' very effectively over time.<br />
<br />
I think the big challenge presented by a 'Lifelogging' device; one which primarily records images and GPS data, is one of data presentation and organization. Is it searchable? Is it relevant? Is it coherent and concise?<br />
<br />
Given this vast array of images and GPS data, it needs to be boiled down into a narrative; one which can be searched at a moments notice.<br />
<br />
What I would like to see is a system which could sift through every image and all GPS data and boil it down to a descriptive text narrative summary, completely automated, and also augmented by annotation from the owner.<br />
<br />
The GPS data is the easiest to tackle, it would be relatively straightforward to take a raw set of GPS data points and convert that into a sequence of 'I went to this restaurant for lunch, stayed about an hour, then drove to the office...' that sort of thing.<br />
<br />
It would be even more compelling if it could say, "Had lunch with my best friend Mike Carlson'..<br />
<br />
How would that last part be possible? Well, we have all seen how photographs are tagged in Facebook. It would be basically the same procedure. At the end of a day review, you could tag pictures of people your Lifelogging device had captured. You could tag the person's name and their relationship to you; wife, brother, co-worker, best friend, etc. Once that person had been tagged a few times, face recognition software would be able to extract that data at a later date automatically. So, you would only have to tag each new person you encounter once or twice, and after that the automated software could take over.<br />
<br />
What's really interesting is this entire process could be done in complete privacy. Only you are seeing the images which you are tagging, and you own; and no other human being.<br />
<br />
So, here is the process I'm suggesting:<br />
<br />
Step 1 : A Lifelogging device captures a days worth of pictures and GPS data. (It would be cool if it captured some biometrics too; but that's certainly not necessary. It would be amusing to see what adjectives could be added to the narrative summary if it had bio-metric data to go along with it.)<br />
<br />
Step 2 : The data gets transferred into the cloud. The cloud processes the data to produce a first draft narrative summary.<br />
<br />
Step 3 : The user has an opportunity to tag pictures and places that the Cloud could not immediately recognize from previous sessions or open source databases. Example, a street address becomes 'Bob's house.'<br />
<br />
Step 4 : A revised and corrected narrative summary is presented; which the user can then embellish, correct, or add to; as a form of micro-journal.<br />
<br />
Step 5 : These narrative summaries can be processed by a Google like knowledge based search engine; so you can later be reminded things like. "The last time you went to this location was on such and such a date. Or, you have been to this place so many times before. Or, "You saw your friend Mike at the office last Friday." All of this, of course, would be fully hyper-linked and searchable. It is this mechanism where the 'photographic memory' begins to take shape. In time; being able to query this in real-time using voice would be truly awesome. The ultimate would be something with all of these features integrated to 'Google Glasses' to such an extent that when you see someone you would immediately be presented with 'memories' of your interaction with them over time.<br />
<br />
Step 6: The narrative data is organized hierarchically. It could break down to be as granular as what you did in the morning, afternoon, evening, next to a daily basis, then a weekly basis, what you did on the weekend, what you did this month, etc. etc. By drilling down hierarchically through the data, from big picture views all of the way down to the detail of a snapshot of a specific event, I believe this is the true promise of 'Lifelogging'.<br />
<br />
I also believe, however, that without this form of hierarchical narrative summary then 'Lifelogging' will just become another useless fad. People will play with it for a few weeks, and then get bored and tired with it. They will stop wearing the device and it will sit gathering a dust in the corner. They might wear it on special occasions, like a vacation or excursion, but give up on daily use.<br />
<br />
A massive pile of uncollated and uncorrelated data will prove completely useless and pointless. This has already been the case with early researchers who have tried this process.<br />
<br />
Some other important features I would like to see in the Cloud service are sharing / privacy settings. The user should be allowed to make some parts of his lifelog visible to his close friends and family. Another feature I would like to see is a way to hyperlink between the lifelog and Facebook posts, YouTube videos, and other content captured through systems and devices not directly related to the Lifelogging device itself.<br />
<br />
It occurred me today, even before I receive my Memoto Lifelogging camera (which I have already pre-ordered) that I already have the capability of capturing a fair amount of data today. Just by virtue of carrying around my cell phone with me, I could easily record my GPS co-ordinates everywhere I go. Probably the easiest way to do this it to simply use <a href="https://latitude.google.com/latitude/b/0">Google's Latitude</a> application. Google Latitude will record everywhere you go and let you review it online using either a web-browser or Google Earth. That's pretty cool, but it lacks the concept of a 'narrative summary' (i.e. I went here, stayed for two hours, then went here..)<br />
<br />
I searched both the Apple app store and the Google Play store and while there are a whole bunch of GPS logging apps; most geared towards hikers, bikers, and runners, none of them yet have a slant towards 'Lifelogging' services.<br />
<br />
Just out of curiosity, I was wondering what would be involved in writing an application myself. It wouldn't be that hard. I can get the GPS data off my phone easily enough, and there is an open source locations database that I could use to convert the co-ordinates into text descriptions of place locations.<br />
<br />
So, it really doesn't seem like it would be that difficult to do. I'm pretty psyched about the 'Memoto' Lifelogging camera; I'll have to make an update once they finally ship the device. <br />
<br />John W. Ratcliffhttp://www.blogger.com/profile/13580494289034594888noreply@blogger.com4