I realized that I had imagined the AI would take in my entire chart and make a diagnosis, possibly with some dramatic gradually-appearing images like the scenes onwhere they discover a large tumor that creates a narrative complication and is solved by the end of the episode. I’vebefore about this phenomenon, where unrealistic Hollywood conceptions of AI can cloud our collective understanding of how AI really works.
Humans use a series of standard tests to generate a diagnosis, and AI is built on top of this diagnostic process. Some of these tests are self-exam, mammography, ultrasound, needle biopsy, genetic testing, or surgical biopsy. Then, you have options for cancer treatments: surgery, radiation, chemotherapy, maintenance drugs. Everyone gets some kind of combination of tests and treatments. I got mammography, ultrasound, needle biopsy, genetic testing, and surgery.
Because Geras and his collaborators pre-trained the model and put it online, all Robinson and I had to do was connect our code to the pre-trained model and run my scans through it. We teed it up, and … nothing. No significant cancer results, nada. Which was strange because I knew there was breast cancer. The doctors had just cut off my entire breast so the cancer wouldn’t kill me.
We investigated. We found a clue in the paper, where the authors write, “We have shown experimentally that it is essential to keep the images at high-resolution.” I realized my image, a screenshot of my mammogram, was low-resolution. A high-resolution image was called for. Robinson discovered an additional problem hidden deep in the image file. My screenshot image appeared black and white to us, like all X-ray images. However, the computer had represented the screenshot as a full color image, also known as an RGB image. Each pixel in a color image has three values: red, green, and blue. Mixing together the values gets you a color, just as with paint. If you make a pixel with 100 units of blue and 100 units of red, you’ll get a purple pixel.