One of the most remarkable things about working in China is how much skill the workers have out there. I think the video below speaks for itself.

[Youtube link for those who cannot view embedded SWF.]

This guy works at the factories that sew the chumby bags. Apparently, he’s not their fastest employee. They have one who is about twice as fast, and he has been with the company for about seven years. I went to his workstation, but when I got there he was already gone to lunch because he had finished everything. And I mean, there were two enormous bins of finished cosmetics cases next to his workstation.

I think it’s also interesting to notice that the guy in the video above is listening to his iPod while he sews.

Another thing that’s pretty amazing is how rubberized tags are made in China. These are the tags you see all over clothes–chances are you are wearing a piece of clothing or you carry around a bag with a tag like this. I always thought that the tags were pressed by a machine.

I was wrong. All those words, colors, and letters–they are drawn by hand.

Amazing.

[Youtube link for those who cannot view embedded SWF.]

I asked PCH if they had any mechanized factories for this kind of stuff. They told me that they exist, but the minimum order quantity is enormous–hundreds of thousands, sometimes millions–because of the extraordinarily low cost of the product and the relatively high cost of the tooling for the automated process.

This is consistent with a comment someone made to me once about the McDonald’s Happy Meal toys. If you look at the bottom of one, it’s held together with screws. That’s because it’s cheaper pay someone to screw together that toy over the whole production run for it than it is to make a steel tool with the necessary precision so that it just snaps together.

There is a similar trade-off inside the Chumby hardware. There are four connectors on the internal chumby electronics. One had a best price of about $1 US, and the other three had a best price of about $0.40 US each–using the US-based vendors that I could source. PCH’s very talented sourcing expert (she has a reputation that is feared and respected by every vendor) managed to find me connectors that cost $0.10US and $0.06US respectively–saving almost a full $2 in cost. There’s one catch: these connectors don’t come with the sacrificial plastic pick and place pad that enables them to be machine-assembled.

The solution? Witness the man below.

On every chumby, he hand-places these connectors, for about a nickel per chumby.

Thanks to him, chumbys are $2 cheaper–which frees up more money for us consumers to spend on $2 coffees at Starbucks.

This story needs a little background to fully appreciate.

There is a microphone in the new chumby. The particular microphone I decided to use has an integral pre-amp FET (an “electret” type). As such, it must be inserted in the correct orientation with respect to the circuit so the FET receives a proper bias current.

The first samples I got back from the factory had the microphone in backwards. So I called the factory and told them that they need to reverse the polarity of the microphone. I was going to come in for a visit the next week, and I wanted to see corrected samples. When I got to the factory and tested the microphone, I found out to my dismay that the microphones were still not working.

How could this be? There are only two ways to put a microphone in.

It turns out that they had two operators on the line assembling the microphone. One solders the red and black wires to the microphone. The next solders these red and black wires onto the circuit board. The operators were told to reverse the order, and both of them dutifully complied…giving me a microphone that was still soldered in backwards, but with the color of the wires swapped. This is actually a pretty typical story for problems in China…

At any rate, this leads up to the real point of this post. The next day, we had a first pilot run of 450 circuit boards scheduled up. Everything had to go perfectly for us to be on schedule. We had stencils rebuilt (we were debugging a yield issue with the QFN packaged audio CODEC as well) and ready by around noon, and around 6PM I had the first boards in my hands to test. As I was running the final factory test, the device failed again–at the microphone.

Needless to say, this was not a happy moment for anybody in the factory, as the factory is liable for any manufacturing defects. I donned my smocks and marched onto the line to start debugging the problem.

That’s me at 3 AM later that day. I’m still in the factory, and so is every manager and tech involved in the project. The pressure was fairly enormous–right next to us was a line churning out 450 potentially defective circuit boards, and I was unwilling to pull the plug on it because I didn’t know what the root cause was yet, and we had to stay on schedule.

I literally had a panel of factory workers standing by me the entire night to help me with anything I needed–soldering irons, test equipment, more boards, X-ray machines, microscopes. The remarkable thing is that not a single one hesitated for a moment, not a single one complained, not a single one lost focus on the problem, people cancelled dinner plans with friends without even batting an eyelash. If they weren’t needed that moment they were busy overseeing other aspects of the project. And this went on until 3 AM. With one exception, I hadn’t seen blind dedication like this since I worked with the autonomous underwater robotics team at MIT.

Embarassingly, the problem wasn’t their fault in the end. It was the new firmware release that was given to me earlier that day by the team in the US–it had a bug that disabled the microphone due to a hack that was accidentally checked into the build tree.

I think even more impressive is that when they found this out, nobody was angry, nobody complained (well, the sales lady gave me a hard time but I felt I deserved it; nevertheless, she was kind enough to accompany me on the production line all night long and be my translator, since my Mandarin isn’t up to snuff). They were simply relieved that it was not their fault. We all parted ways and I came back into the factory the next day at 11 AM after a good nights sleep. I saw Christy and I asked her when she came in. She told me she always has to report in by 8 AM. Now, I was starting to feel really bad–she stayed up late because of our bug and she came in early while I slept in. I asked her why she stayed up so late even though she knew she had to report to work at 8 AM–she could have gone home and we could have continued the next day. She just smiled and said “it’s my job to make sure this gets done, and I want to do a good job.”

On the right is Christy (PM), and to her left is Xiao Li (QA manager). The equipment they are observing is the chumby production tester, a bed-of-nails device developed in the US by me that facilitates the automatic firmware programming, unique keying, and testing of every circuit board.

Here’s another interesting story. On our way out of the factory floor one day, Xiao Li asked me what does a chumby do? Well, I don’t speak chinese very well, and she doesn’t speak english very well either, so I decided to start with a few basic questions.

I asked her if she knew what the world wide web was. She said no.

I asked her if she knew what the internet was. She said no.

I was stunned. Here is a girl who is an expert in building and testing computers–I mean, on some projects she has probably built PCs and booted Windows XP a hundred thousand times over and over again (god knows I heard that darn startup sound a zillion times that night on the factory floor, as right next to me was a bank of final test stations for ASUS motherboards)–yet she didn’t know what the internet was. I had taken it for granted that if you touched a computer today, you were also blessed by the bounties of the internet. I felt like a bit of a spoiled snob and a pig all at once for forgetting that she probably couldn’t afford a computer, much less broadband internet access. If she were given the opportunity, she was certainly smart enough to learn it all, but she’s busy making money that she’s probably sending back to her family at home.

How do you describe the color blue to the blind? In the end, the best I could do was to tell her it was a device for playing games.

Around 2,500 years ago, the phrase “min yi shi wei tian” was coined by Prime Minister Guan Zhong; there are several ways to translate it. One side of the coin takes the literal approach and says that “people consider food divine”, or “for people, food is next to heaven”. The other side of the coin looks at it as a piece of governing advice, “the government’s mandate [synonymous with heaven] is only as robust as the food on people’s plates”. Or, it could be just thought of as an excuse to procrastinate: “let’s eat first [since it is as important as heaven]”.

Which ever way you cut it, I think the saying still holds in China. So one important metric for gauging the quality of treatment of the factory employees is how good the food is, as it’s common for factory workers to be housed, fed, and cared for on site.

The food is actually quite good at some factories. The photo above is at the factory that does the chumby circuit boards. It was a mix of steamed fish, broiled pork, egg rolls, clean fried vegetables, and some pickled-vegetable-and-meat combo. Rice, soup, and apples were also provided in help yourself quantities.

One interesting fact is that every facility I went to had separate utensils and plates for guests. You can see in the above photo how my food is on a styrofoam plate with disposable chopsticks, where as the factory worker’s food is served on a steel plate with steel chopsticks. This is because I haven’t passed the factory’s physical examination. They do this to prevent me from contaminating the factory with potential foreign diseases. The food in the above photo consists of smoked fish, ma po tofu, boiled vegetables, and turtle shell jelly (for dessert). The turtle shell is very bitter and apparently it’s supposed to make you feel cooler on the hot summer days in China.

The menu is quite interesting in China. I think the menu the day I went to this factory had items on it like pig intestines, kidneys, fungus and vegetables, along with some other more western-friendly items.

Again, the scale of some food operations is pretty impressive. I heard that Foxconn–the place that makes the iPods and iPhones–consumes 3,000 pigs a day. I saw this truck of pigs going off the exit on the highway toward Foxconn, and it reminded me of that factoid.

From pigs to iPhones! It all happens right here in Shenzhen.

I found setting up a supply chain in China to be enlightening, and I’d like to share more of my experiences with the readers of this blog.

Presenting this material is challenging; there is a lot of detail and its easy to get overwhelmed and lost in the vagaries of electronics manufacturing. In order to focus my writing and create a more succinct body of works, this series of posts will be thematic. In other words, instead of talking about the details of how pick and place machines work in China, I’ll post about a more abstract theme such as “automation” and just show sound bites to give the reader a flavor of what’s going on.

The theme of this first post is “Getting Started”.

In September of 2006, Chumby was just a team of about a half dozen people, and we had given away about 200 early prototype chumby devices at Tim O’Reilly’s FOO camp. The devices were well received by the attendees of the conference, so I got the go-ahead to build the Asian supply chain.

Before we went to China, we had a trusted vendor in the US give us their best quotation for the job, so we had a reference point to work downwards from.

In a nutshell, we called up a lot of our friends who had some experience in China, and we lined up a series of about a half dozen factory tours in China. We hit quite a variety of places on our tour, from specialty factories as small as 500 people to mega-factories with over 40,000 people.

Being open source helped a lot in terms of kick-starting the process, because first we had no fears about people stealing our design–we’re giving the design away–and there were no NDAs to sign when sharing critical information, like the bill of materials. I think this kind of openness actually gave us a better reception with the factories in China; they seemed more willing to open up to us because we were willing to open up to them. Also, there was no question in the factory’s mind that they were in a competitive situation. It was plain that anybody could and would quote and bid on the job (in fact, we received a few unsolicited quotations that were quite competitive), so it saved a round of huffing and puffing. And finally, it seems that despite handing out our BoM without an NDA, nobody in the general public is aware of the strategic improvements we rolled into the chumby hardware since the FOO prototype (then again, chumby has no serious competition at this point in time).

In November, Steve Tomlin and I went to China to do the factory visits, and some of my first impressions were documented in my post Adventures with the Venture Communist.

There is no substitute for going to China to tour the factory. Pictures are always deceiving, and you just can’t get a sense for the scale and quality of the factory without going to see the actual facility. In general, factories welcome you to take a tour–I wouldn’t work with one that didn’t allow me to come visit. However, most factories do appreciate a week prior notice before you come and visit, although as your relationship with the factory progresses things should become more open and transparent.

While almost every factory will “clean up” the day you come to visit, a sharp eye and the right questions can see through any quick veneers put in place. One place I always liked to visit was the QC room. I expect to see rows of well-maintained and well-worn binders with design documentation and QC standards, as well as “golden samples”. I would point at a couple of random binders and demand to view their contents, see the product’s golden sample, and verify that they knew what was going on in the binder–that it wasn’t just some random data they threw in there. Also, hard investments in equipment is a good sign: the best manufacturers I visited all had a couple rooms with sophisticated equipment for thermal, mechanical, and electrical limit testing, and of course operators were in the room actually using the equipment (I could definitely believe a Chinese manufacturer would buy a room of equipment just for show and not actually use it).

After reviewing several manufacturing options, we decided on one, which was PCH China Solutions. PCH itself owns only a few facilities, but it has a comprehensive network of trusted and validated vendors–primarily in China but also Europe and the US as well. Not surprisingly, the factories that PCH sub-contracts to were some of the best facilities we visited while we were in China.

PCH is actually headquartered out of Ireland, and as such, most of their staff engineers are Irish, so there was no langauge barrier. They are hard working, resourceful, well-trained, and as a bonus they always seemed to know the best places to find a pint no matter where we are. I had no idea China had so many Guiness taps.

In retrospect, this is definitely the way to go–I learned a lot about what it takes to do business in China by shadowing their engineers while I was out there. From 10,000 miles away it looks easy but it’s anything but easy working with even the best Chinese contract manufacturers. I’ll cover some of the “frustration stories” in another themed post later on.