[EEG] From the Conception to the Closing of an Earphone-type EEG Measurement Device

Declaration of Retreat

Although I had a sense under the skin that the service development was quite challenging, I also thought that I would soon get bored if I did not challenge myself with this level of difficulty, so I decided to proceed. In order to cut off the retreat, I made a declaration on twitter.

https://twitter.com/kazu_098/status/1436315493569208322

As we proceeded with the overseas development of MERP, I had the feeling that we were too late in expanding overseas, and I knew that the next step was to release our products globally from the beginning. In order to do so, I had to create services in English from the beginning, and although I could do that while in Japan, I felt that if I stayed in Japan, I would inevitably end up looking for Japanese engineers and releasing services in Japanese. I started to think about moving to the U.S.

At the time, there were only about 200 or so hits on earphone EEG when searching for papers, and among these, the only companies that were developing their own products were a Danish professor who had been researching earphone EEG for over 15 years and the UC Berkeley professor who wrote the aforementioned paper. I thought it would be ideal if I could join the Berkeley lab as a graduate student or research student and conduct experiments and product development at the same time.

In fact, I DM’d the professor on LinkedIn with a resume of the product I wanted to create, but since I had no original contact with him, he didn’t go through with it, so I decided to proceed with the development by myself once in Japan (I later learned that the professor himself had launched a startup with this service).

Where do I start?

Although I understood the general concept from reading the paper, I have no basic knowledge of the specific method of creating the electrodes described in Method or around electronic circuits, so I have no idea what it says at the terminology level to begin with.
I wondered how exactly to proceed, so I consulted with an acquaintance who has already developed earphones, and he suggested that I first attach conductive materials such as aluminum or copper to existing earphones and measure them. I decided to experiment. This is the prototype.

However, since this is only an electrode, an electronic circuit board equipped with an amplifier to amplify the brain waves (electrical signals) that flow through the electrodes, an ADC to change analog signals into digital signals, and a Bluetooth function to send the signals to the PC side are separately required. As for this, I found a device called OpenBCI, which is already commercially available and can display brain waves on a PC as a GUI, so I decided to purchase it, connect it, and experiment with it.

When I look at it now, the price has gone up even more to about 150,000 (it was about 80,000 when I bought it), but still, considering that EEG meters used in laboratories are highly accurate but cost 1 million or more, I bought it as it is somewhat easy and shortens my time (now that I know the specs inside, I can’t help feeling I am making a ragged profit. I think there is totally a business of replacing this area by offering smaller, lower price range, higher spec EEG meters).

I didn’t expect to get it from the beginning, but as expected, I didn’t get it at all.
This is where the trial-and-error process began, and also where we fell under the spell of OpenBCI.

The Curse of OpenBCI and its Conversion to Polymate

The development of brain wave devices, even at the prototype stage, covers a wide range of areas in hardware and software, including 3D printers, consideration of electrode materials, analog electronic circuits, signal processing, BLE communications, and software programming. So I asked around to my acquaintances to see if there were any engineers who could do the hardware side of the project.

Then, I happened to connect with a person who happened to be from the same hometown, with whom I had mutual acquaintances, and who had been doing electronics work as a hobby since junior high school, and we decided to work together. He had also studied myoelectricity in college, so we were a perfect match.

I think it was around October 2021 when we met, and from that point on, I spent my weekdays doing merp and ishiyak, and my weekends going to that person’s house and just repeating development and experimentation. This encounter made it possible for me to quickly turn the cycle of hypothesis testing in my experiments.

At that time, I trusted the OpenBCI specs completely, and without questioning that side, I was just experimenting mainly around the electrodes and the shape of the earphones. Since I did not have much knowledge of the hardware side, I could not question them in the first place, and there were papers that announced that they were obtained using OpenBCI, so I did not particularly doubt them (I would have lost a lot of time here).

This is what the prototype looked like at that time.

Although there is another problem that the silver spray does not adhere to the silicone and peels off, he is unable to measure EEG with OpenBCI only in the ear, and gradually suspects that the OpenBCI is bad. He began to suspect that the OpenBCI might be bad, and began to consider purchasing a different EEG measuring device.

In fact, OpenBCI works fine with wet electrodes with gel, but with dry electrodes, as in this case, it had a drawback that it did not work well as it was due to skin electrode resistance, but it took me a while to realize it.

At the same time, in light of international patents, we also decided to proceed with a patent application, although it was still on an idea basis, with a patent attorney who had been helping us since the MERP project.

Also, there were relatively few papers on attempts to acquire EEG from earphones, so I read all the important ones and summarized them in NOTION.

What about analog electronic circuits?

I had no idea what electronic circuits were. I had done software programming before, but I thought that hardware might be difficult. I had assumed that I had experience in software programming, but hardware programming might be difficult, so I decided to look for someone who could do it.

However, we found that these days, the circuits are becoming more digital and fewer and fewer people are familiar with analog circuits. He said that there is no one who knows much about analog circuits unless he is a professional in his 50s or 60s who works for a manufacturer.
At first, I tried calling out to people on twitter and FB with no luck, but I couldn’t gather anyone, so I decided to start studying because I knew I had to understand it myself.

In retrospect, this decision should have been made earlier. I had a preconceived notion that it was somewhat difficult.
I read the following book, which was recommended by an acquaintance who is developing IoT services, to understand the behavior of operational amplifiers, in particular, and bought an electronics construction kit and started creating simple circuits because it is faster to learn by doing.

At first, I thought it was impossible, but gradually I began to understand, and as I did so, I also began to understand the part of the EEG-related paper that explained the electronic circuit. When I actually understood, I found that the EEG circuit did not do anything so difficult, but only extracted and amplified the desired signal with several filter circuits, and the rest landed on the problem of how to select the right op-amp for the EEG measurement.

He also recently learned that there is software (such as LTSpice) that allows you to simulate the circuit on your PC before assembling the components, and started studying that as well. After that, the process of ordering the circuit board, installing the components, and checking them on the actual device is done, and I have managed to catch up by studying that as well, and if I have any trouble, I ask someone in the know or post on a forum.

I was deeply grateful that there were people who immediately gave me kind and informative answers, regardless of race. It was also significant that I was able to experience with my own hands the sequence of infrastructure development for the prototype: “circuit creation, preliminary simulation, order to factory, mounting of components on the received board, and operation check on the actual device”, and the electronic circuit for EEG measurement was once completed (I think).

Keio University Faculty of Science and Engineering, meeting with Prof. Miki

This may sound a bit technical, but brain waves are not something amazing, but simply electrical signals, so when measuring from the surface of the body, an electrically conductive substance must be glued to the skin to energize it and send the brain waves to the measurement device.

However, it has to pass through the skull to skin layer before reaching the skin, so it is much attenuated when it reaches the surface of the skin. In particular, there is significant resistance in the outermost layer of the epidermis, called the stratum corneum, and the EEG signal is weakened in that area. One approach is to prick the area with a needle to reduce the resistance and obtain brain waves with less noise (MEMS: abbreviation of Micro Electro Mechanical Systems).

In the course of experimenting with OpenBCI, I still could not obtain EEG as in the paper, so I searched for other methods and learned that the above approach existed and that Professor Miki of the Faculty of Science and Technology at Keio University, my alma mater, was conducting research in Japan.

https://top-researchers.com/?p=1366

I sent an email directly to him, to no avail, and to my surprise, he responded immediately and we decided to have an online meeting. Keio has been conducting a medical-engineering collaboration project (medical and engineering collaborate to support startups) for several years, and Dr. Miki, who is also a representative of the engineering faculty, has a deep understanding of startups.
When I gave a presentation on the earphone EEG project, he was very interested in the project and we decided to collaborate.

When one hears the word “needle puncture,” one would expect to be preceded by the image of pain and fear, but in reality, the stratum corneum is composed of dead cells, like the white part of a fingernail, so even if a needle is inserted, theoretically, one would not feel any pain. Nevertheless, it may be unacceptable as a device. Even if it were possible to take brain waves with MEMS technology, the design of the mechanism and the way it is presented to the user would be very important, I thought as I continued my experiments. I will also work on the development of a dry electrode that is not a needle type in parallel, and after January 2022, I will commute to Keio’s Yagami campus on the weekends.

I had only attended Keio in Hiyoshi campus during my first year of college, and I felt nostalgic in Hiyoshi after 15 years. I experimented on Saturdays and Sundays, and repeatedly ate ramen noodles in Hiyoshi for lunch.

First successful experiment!

At least two electrodes are needed to measure EEG, and the difference between the EEG signals measured from the two electrodes is amplified and acquired. The electrical signal acquired from one electrode should contain EEG and the other should contain no EEG at all.

In the course of the experiment, we succeeded in measuring alpha waves when one electrode (the detection electrode) was placed in the ear and the other electrode (called the reference electrode) was placed on the mastoid process (the protruding part behind the ear), although not the type with all electrodes in the ear.

I was so pleased at this point that I could just keep all the electrodes in my ears and alpha wave detection would soon be available! With alpha wave detection, specifically, it would be possible to score meditation, alert you when you doze off during a meeting, for example, OR automatically change the music to classical music that promotes relaxation, etc.

In fact, at that time, we had switched to Polymate, a high-performance EEG measurement device that is also compatible with dry electrodes, instead of the OpenBCI electroencephalograph device that we had mentioned. If we had used this electroencephalograph device from the beginning, we would have been able to shorten the period by 3-4 months, but there was nothing we could do about this.

https://www.miyuki-net.co.jp/jp/polymate/

Eventually, the electroencephalograph itself would have to have its own EEG circuit, not an external one, and also have to be housed inside the ear, but once the experiment was underway, the electrode area and then the EEG measurement circuit area were separated.

UC Berkeley’s Accelerator Adopted by SkyDeck

This was really one of the major turning points for me. In my next start-up, I wanted to expand globally from the beginning, so I wanted to move to the U.S. for business. However, there were visa issues and the prototype had not yet been completed, so I did not know how exactly to proceed or how to establish a foothold in the market, so I was developing the product in Japan.

He replied, “There are only a few days left until the deadline, but JETRO is running an acceleration program to support startups’ overseas expansion, and I participated in it last year. I was glad to have participated in that program last year. I actually established a U.S. corporation and obtained a visa,” he replied.

We also needed a website for the service, so we created the following in a rush with Strikingly

https://ear-brain.mystrikingly.com/

I have rarely been accepted for these types of document-based programs (I have been wiped out for overseas master’s programs and scholarships in the past), and I really did not expect to be accepted, and when I received the email, I almost jumped out of my skin.
Moreover, it was SkyDeck, a startup program at UC Berkeley, which I had submitted as my first choice.

https://prtimes.jp/main/html/rd/p/000000124.000071241.html

The program itself will last approximately three months, from November 2022 to February 2023, and the final DemoDay will be attended by nearly 100 U.S. investors. If investors become interested in the program at that pitch, it will serve as a foothold for business development in the U.S.
Incidentally, the selected companies were expected to go there during the first half of the program, from November to mid-December, so I hurriedly arranged my travel tickets. I also decided (and had to quit) the once-weekly part-time job as a physician’s internist and visiting doctor that I had been continuing until now at the end of October, and decided to go solely on the business.

The density of the ecosystem surrounding startups is different.

Arriving in San Francisco on Nov. 1 with some anxiety and a lot of anticipation, I booked a 6-week stay on Airbnb and was surprised to find that the rent for two people, including the co-founder, was more than 600,000 yen. The rent was about three times as much as in Tokyo due to the weak yen. It is too expensive. Eating out for lunch was about 2,500 yen, about 3 times the price in Japan.

Jun, the SkyDeck program coordinator, was very helpful, quick to respond, and personable.

This year’s SkyDeck is the 15th edition and is called Batch 15. More than 20 companies from Japan via Jetro and 120 companies from around the world are participating. At the first orientation, Program Director Caroline gave an overview of the program.

This Caroline is a super powerful woman who was originally a professional violinist, but has since started and exited two companies and is currently a triathlete and has completed an Ironman. Her body shape and tone of voice seem to exude stoicism. Just as a person who has never founded a business before would give one-sided advice on startups, the fact that the Program Director herself is an entrepreneur who has already exited multiple times made me feel that she is indeed the world’s top accelerator.

The three-month program is divided into three main parts: the first month consists of lectures by instructors on company formation, how to build a business, marketing, recruiting, fundraising, etc. In the second month, each company is assigned a mentor to help them finalize their business plan. In the final month, the students are given weekly instruction on how to make a presentation for the Demo Day.

Personally, I felt that there was no difference in the competence level of the entrepreneurs themselves, but the density of the ecosystem surrounding entrepreneurs and startups was very different from Japan.
For example, the lecturers often talked about general topics (e.g., “select and focus on products at the beginning,” “focus on the narrowest target possible,” etc.), but the lecturers themselves had started multiple businesses and had IPOs, and there were many serial entrepreneurs, so I naturally felt that this was the norm and my perspective was raised. This naturally raises one’s perspective. In addition, since a network of advisors has already been established for each of the areas such as patents, company establishment, and fundraising, there is no need to search for advisors from scratch by ourselves. If we tell the program coordinator what we are having trouble with, he or she will immediately connect us with a good advisor, so we can solve our problems quickly.

Also, after all, what I felt when I came here was that it is normal to create services in English and release them from the beginning targeting the whole world, and I actually witnessed what I had somehow felt for almost a year now, and by placing myself in such an environment, I no longer feel any resistance at all.

I asked Jun if he could somehow get in touch with a professor at UC Berkeley who had published a paper on in-ear EEG. He told us that he could not help us because he himself had his own start-up company and we would be competing with him.
Unlike in Japan, in the U.S., professors often start their own startups, so I had a vague feeling that this was a possibility, but I was disappointed to find out that this was in fact the case.

The program started and I had an opportunity to give a 2-minute pitch about my service in English in front of all the participants in the first week of the program. When I actually presented the service (which is an earphone-type device that measures brain waves and automatically improves sleep quality), I felt that the response was very positive, perhaps because it is an easy-to-understand service.

Many teams of native English speakers participated and gave presentations, but just as it is difficult to convey the concept of a service in Japanese if it is difficult, it is not always easy to convey the service in Japanese just because the speaker is a native English speaker, and some of them were pointed out in their presentations. Some of them were pointed out in their presentations. On the other hand, there were also presenters who tried to cram all the information into two minutes and spoke too fast, making it even more difficult to understand what they were saying.

I had somehow thought that if I were to present in English, I might be at a disadvantage if I was not a native speaker, but as I watched other people’s presentations and listened to feedback from the mentors, I realized that whether or not English was my native language did not matter at all. Furthermore, I thought that because I am a non-native speaker, I cannot speak as fast as I can in my native language, so I have to explain my services slowly, which has the advantage of being easier to understand because I do not speak too fast.
On the other hand, if I were to explain in Japanese, I would speak too fast, and I might be pointed out to speak more slowly.

The service itself was very interesting and responsive, so it was very beneficial to realize that English is not a hurdle to developing business at all.

There was also a networking event for entrepreneurs, and I became friends with almost all of the Japanese entrepreneurs who came to participate with me, and I had lunch and dinner with participants from other countries, which was both stimulating and fun.

Petit-pivot of services to custom earbuds

Until then, EEG measurements had been attempted by attaching electrodes to general-purpose earphones such as those shown below, but with this method, the shape of the ear differs too much from person to person, so the electrodes may not hit properly, resulting in a less stable EEG.

However, one day, I suddenly thought, “I guess it would be a complete turnaround unless the electrodes are placed firmly on the ear to increase the accuracy of EEG measurement. I thought, “Why not make custom earphones that fit the user’s individual ears instead of general-purpose earphones? I thought, “If we don’t make the EEG measurement more accurate, it would be a complete turnaround.

Once the above concept is decided, the development method changes accordingly. For example, how can we easily create earphone shapes to fit the user’s ear shape? and how do you have a mass production system? and other additional questions came up, and the development method became more specific.

The method we were (and still are) considering then was the following.
Users would first download our app and take at least 20 photos of their ears from all angles. Then, using our 3D scanning technology, the shape of the user’s ear is automatically recreated on the app.
However, this is only the shape of the user’s ear, and we must actually create earphones that fit this ear. The user’s ear is the female ear, and the actual earphones are the male ear.

Then, once the earphone casing (shell) has been created, the EEG measurement components are packed inside and mailed to the user.

We thought that by doing this, we could measure brain waves with less noise, since the electrodes could be pressed against the individual user’s ear. With this idea, we really thought that next time we would be able to measure clean EEG.

Incidentally, taking this approach would pose additional challenges, especially in the process of creating the earphone housing.

(No. 1) Can the 3D scanning technology of a smartphone camera really reproduce a 3D copy of a user’s ear with high accuracy?

(No. 2) Even if we could reproduce a 3D copy of the user’s ear, can we automate the subsequent process of creating the earphone housing?
We do not need all of the scanned ear shape, in fact, we only need to be able to create earphones that fit into a concave part of the auricle, so we have to segment the desired image.

The added difficulty associated with the change in approach, measuring brain waves with custom earphones, is one that we thought could be overcome. In fact, two companies in the world, as far as we know, have already put this part to practical use, one of them being an Australian company called Hearables 3D.

https://www.hearables3d.com/

As a side note, if this can be done, music enthusiasts who are particular about sound quality and hearing aids can request custom earphones to be made, but this is actually a process from ear mold collection to the actual earphone creation, which is done almost entirely by hand, and that makes it expensive, ranging from 100,000 to 300,000 yen. However, the ability to automate this entire process will lead to a revolution in the industry as a stand-alone service.

Reunited with Dmm.make Akiba

While participating (and still participating) in SkyDeck, I was receiving feedback from various people, especially on the marketing and business side of the service.

However, although UC Berkeley has facilities for hardware development, they are not readily available to start-ups from outside the university (although they are available, it was difficult to get the equipment I wanted to use in a short period of time), so the service development stopped. I had originally planned to stay until mid-December 2022, but I left early and returned to Japan on November 25.

To speed up the hardware development, I joined DMM.make AKIBA, a coworking space in Akihabara, which I had been a part of when I was a student, on the following day. Looking back now, I wonder why I didn’t join from the beginning, but somehow it slipped my mind.

https://akiba.dmm.com/

DMM.make AKIBA has not only the facilities necessary for hardware development, but also tech staff who are available for consultation if you need help with technical aspects.

The part of the process that uses a smartphone app to scan the ear shape and automatically generate 3D data can be substituted by another 3D scanner machine, so I took a course at AKIBA and immediately scanned my own ears in 3D to create custom earphones that fit my ears. I also ordered an assortment of possible candidate materials such as silver, copper, nickel, carbon, and PEDOT:PSS to test conductive electrodes.

Then, I managed to get alpha waves only with dry electrodes by giving copper-nickel sheet electrodes to my ears. However, it was not reproducible, and only the co-founder, not me, was able to get them, and they were not stable. I began to think that it might be difficult to lower the skin electrode resistance any further because the properties of the stratum corneum, especially of the human ear, are so different.

What I also understood after conducting the experiment was that I initially thought that brain waves were often emitted only in the external auditory canal (ear canal) and almost no brain waves were emitted in the auricle or earlobe, but this was not the case, and in fact, there were as many brain waves in the auricle as in the external auditory canal. As described earlier, EEG amplifies the difference between the electrical signals from two electrodes, and since the distance is too close around the ear and EEG is normally emitted from both electrodes, it was found that they could be detected on their own, but that they canceled each other out and eventually could not be detected.

This could be solved by switching to a method of acquiring brain waves from both ears instead of from one ear, and this was actually tested, but this would require connecting both ears with wires, which is a far cry from the originally planned concept of Comfortable earphones that can be worn at any time. I was wondering how to proceed. We were left wondering how to proceed.

Meeting with Iain