The last step was to fully integrate the media viewer with OpenMRS and Moca. Now the media viewer supports thumbnail handling by performing an HTTP post to an OpenMRS servlet to retrieve the image and downscale it properly. If it is an audio or video file, a preset thumbnail will appear.

I also changed the bottom toolbar to have the sliders stacked on top of each other so that it would be less wide on the screen when added next to the Moca panel. There are 2 new buttons to the toolbar - revert to default image and full screen mode.

Even though I used a pre-made Pan/Zoom component from Adobe Developer Connection, I had to read through the library to understand how to modify it for OpenMRS’ purposes. I added support for each Generic File in this encounter to save the image processing and zoom settings added so that if the user moved to the next thumbnail, the settings would still be saved when the user returned. However, these settings are not permanently saved to the OpenMRS server. According to my usability tests, the specialist wants to manipulate the image but doesn’t need to save the modified version. Another problem was that the image was animated to stick to one of the four corners of the content rectangle. This caused the image to fly to unexpected positions and the mouse would lose the image underneath it (see below). I wanted the ability to pan and drag the image anywhere, so I removed many of their animations and edge boundary restrictions.

With my modifications, the panning is much more controlled and the image stays under the mouse.

Here is the final version of the media viewer! (Although more modifications are coming soon..) Within the Moca module in OpenMRS, the doctor will access a list of pending cases from healthcare workers.

When a doctor clicks on an item, the media viewer will appear.

The doctor can browse through all the media files associated with this patient’s encounter and then fill in diagnosis and treatment recommendations on the left hand side. When the doctor hits the “send” button, it will get uploaded to the OpenMRS server and modify the encounter. The encounter viewer already built into OpenMRS shows the objects already associated with that encounter.

You can see that a new Obs was added to the list for this encounter of concept type “Diagnosis.” The value of the Obs is the text that we entered in the Moca panel above. This view also shows that Dr. Mark Johnson (the user account currently logged into OpenMRS) made this diagnosis on August 25. Different doctors are allowed to make a diagnosis for an encounter and a single doctor can also make multiple diagnoses if they change their mind.

Here are more examples of how the media viewer can be used in other applications. For this tele-oral case, you can add a summary below the image.

Or pop out the summary to move it around and allow you to zoom in on the image closer.

For teleradiology cases, many patients will have multiple x-ray views taken to help the doctor with the diagnosis. These can all be viewed together easily in the workflow. The 4 x-rays for this patient are displayed in preview thumbnails along the bottom.

You can turn off the toolbar on the bottom for a bigger view of the image. (Use the options menu in the upper right corner).

If you try to fix the image (brightness, contrast, sharpness, zoom) but don’t think the settings improve the image, you can always revert to the original image with this button on the toolbar.

The original image is back!

If the encounter contains an audio file, simply click on the audio thumbnail along the bottom toolbar and the file will begin playing. You can play/pause at any time, mute/unmute, change the volume, and skip track position. The audio player also displays the current time out of the total time length of the file, which provides the doctor with a cue point in annotating the file.

Here is an example of video playing in the media viewer. This is a video taken by the G1 phone in the MIT media lab. It was converted to FLV format using the Adobe Media Encoder. You can play/pause, mute/unmute, change the volume, skip to any position in the video, and change to full screen mode.

Example of ultrasound video being played in the media viewer.

Video full screen mode!

That’s it! Next features to come: drop down boxes for diagnosis/intepretation fields (for data mining purposes), upload a file to the encounter viewer, and download all files.

Unfortunately, I realized that Flex only supports MP3 audio formats, as well as FLV and H.264 encoded videos. This would be an obstacle, since we wanted the media viewer to be able to support all types of audio and video. (Then doctors would be able to upload medical files freely without worrying about compatibility). In addition, the project was designed with the desire to upload video from mobile phones into OpenMRS.

Specs for what Flex supports:

I captured some sample videos on the Android G1 phone to see if it could be embedded in my Flash application, but the video didn’t play properly. It turns out the problem was because the video was in .3gp format and using H.263 encoding. You can find encoding information using Ctl + I when viewing the video in Quicktime (there are no menu options for this).

I double-checked the Android documentation and found this page to confirm that audio and video files created by the Android G1 phone are NOT in MP3 or H.264 format. (Source: http://developer.android.com/guide/topics/media/index.html)

After discussing this format support problem in Flex with other computer science students at MIT, I came up with two design options: 1) Find an open source video player that supported crucial formats (i.e. AVI, MP4) and embed that using HTML/Javascript in my Flash application, or 2) Convert all files uploaded to OpenMRS into MP3 and FLV format to be played by the Flash media viewer. I tried looking for a media player written in Flex so that it could be easily integrated with my application. I found various players, but the FlowPlayer (http://flowplayer.org/demos/installation/index.html) looked the most promising. However, after testing it with my sample MP4 files, sometimes the audio would play without the video portion. I also had difficulty importing the player into FlexBuilder because it required Adobe CS4 Professional.

After much more research and experimenting, I decided to go with the second option of requiring the format to be MP3 or FLV. This was a better idea because raw video files can take over a hundred MB and would provide variable output quality/size. Meanwhile, storing the videos in FLV format would only take up 10-15MB of space and would stream much faster when the doctor loaded the page. (Youtube.com also converts all uploaded videos into FLV format for streaming).

Hence, I looked into ready-made programs that would perform video transcoding for me. There are a lot of options on the Internet but many charged money for the solution or had a GUI interface (so integrating it to run automatically would be difficult). A command-line option that looked most promising was FFMPEG (http://ffmpeg.org/). FFMPEG is an open source cross platform solution to streaming and converting both audio and video. It worked on converting many of my MP4 test files into proper FLV files, but it didn’t support conversion of 3gp files form the phone. If you’re interested in using this tool for your project, this tutorial is very easy to follow and understand.

FLV Converter - A Layman’s Guide to Flash Video Conversion

http://worldtv.com/blog/guides_tutorials/flv_converter.php

Then I looked into Adobe solutions and found the Adobe Media Encoder. As I mentioned above, Adobe CS4 Professional requires a license, but I decided to just use the trial version since it includes the Adobe Media Encoder. It worked very well and converted all my files to  FLV format, including my 3gp test file. There’s also a command-line interface for the media encoder (http://help.adobe.com/en_US/FlashMedia-Encoder/2/help.html?content=get_started_11.html), so it can be integrated with the Moca module on OpenMRS. However, the only problem left is that we would need a license for the software. I decided to assume that transcoding would be an issue that we would deal with later, at the very least, the user could convert the file using a free converter from the Internet. I moved onto spend time on developing the audio and video player for my media viewer.

Flex doesn’t have any pre-built media players for easy integration into projects, so I had to browse online for ones made by other developers. After much searching, I didn’t find any players that suited the needs of the OpenMRS media viewer. For example, the audio players were too simple (just play and mute buttons), too complex (with filters and audio synthesis tools), or didn’t have open source licenses. As for video players, I found ones like Flow Player (described above), but then realized that there is an FLVPlayback component in CS4. I imported the SWC from CS4 into FlexBuilder and used this pre-made component for my media viewer.

Using the Flash FLVPlayback control in Flex

http://blog.flexexamples.com/2008/12/11/using-the-flash-flvplayback-control-in-flex/

Building Web Video Players

http://www.adobe.com/devnet/video/customization.html

FLVPlayback Directly in Flex

http://yourpalmark.com/2008/04/30/flvplayback-directly-in-flex/

I modified the version from this tutorial to a different skin with more control options, docked the control bar below the video (no fading), made audio/video play simultaneously, and changed my Flex application HTML file to enable full screen mode. Pictures of the finished FLV player will be in an upcoming post.

Since the FLV Playback component didn’t play MP3 files, I had to come up with an audio player that somewhat visually matched the video player. There were many tutorials on how to use the Sound and SoundChannel classes in Flex, so I decided to create my own audio player.

Creating an audio player turned out to be a challenging task even though there were pre-built Sound classes.  Some difficult parts were figuring out to update the time and slider as the audio played, allowing the user to skip to a different position in the track, switching the play icon to a pause icon (as well as mute/unmute icons), linking the volume control slider with volume changes in the sound channe.

Foundation Actionscript 3.0 with Flash CS3 and Flex

http://books.google.com/books?id=B4aB0hJsNMoC&pg=RA1-PA319&lpg=RA1-PA319&dq=track+sound+position+flex&source=bl&ots=t-W4JrGLqK&sig=ikrulCJ6pcQ0LoE3RU6w1Bu9o2c&hl=en&ei=dCCSSsnwFpSnlAe29pmiDA&sa=X&oi=book_result&ct=result&resnum=1#v=onepage&q=&f=false

Working with Sound

http://livedocs.adobe.com/flex/3/html/help.html?content=Working_with_Sound_12.html

Loading External .mp3 File and Retrieving ID3 Information

http://manewc.com/2008/02/07/as-3-loading-external-mp3-file-and-retrieving-id3-information/

Create a Simple and Attractive MP3 Player

http://flash.tutsplus.com/tutorials/actionscript/create-a-simple-and-attractive-mp3-player/

Coming up with the button skins and look-and-feel of the audio player took some time. Here are some design iterations:

The goal is to embed the OpenMRS media viewer for encounters into the Moca module. Hence, when healthcare workers upload patient symptoms, images, audio, and video, the doctor can login to OpenMRS and view all this information within one screen for a smooth workflow.

Moca consists of a queue of pending cases from healthcare workers, and when the doctor clicks on one item, he/she will be brought to a page that contains a panel of patient symptoms and diagnosis options (on the left side) and the Flash media viewer (on the right side).

To make this panel for Moca in OpenMRS, I used HTML, CSS, and JSP. It displays the patient name and links to his/her patient dashboard in case the doctor wants to look up information on past patient visits. Then it lists question/answer response pairs according to the procedure defined by the doctor and downloaded to the G1 Android phone. These questions include “What is the size of the lesion?” or “Does anyone else in the family have this condition?” Although it would be easier to display the entire list on the screen at one time, I decided to use a scrolling box for scability because some of the procedures can get very long.

For the section where the doctor can write a diagnosis, I originally had it set up as the following (see below). This panel was designed using feedback from the doctors in the Batanes (in the Philippines). The radiologist said that he only provided findings/interpretation and other comments for an x-ray. He didn’t assign a diagnosis or treatment (that was the clinician’s role). However, for the teledermatology cases in Africa with Click Diagnostics, the specialists would provide a diagnosis and treatment. Because each pilot would involve different specialists with different medical applications, with the help of my team, I decided to instead make this panel customizable for each workgroup or pilot. Hence, radiologists in Manila would require 2 fields (findings/intepretation and other notes), while doctors in Africa for dermatology and opthamology would use 3 fields (diagnosis, treatment, and other notes).  I’m working on this custom interface, and will write about this soon.

The other parts of this interface came from my usability tests on doctors in the Philippines. For example, they wanted to see the name of the doctor writing the diagnosis to ensure proper identification of who made the diagnosis. Other options they wanted were to send a reply to the healthcare worker asking for a retake of the image and the ability to leave the case pending or to refer it to another specialist.

Taking all of these workflow requirements into account, I came up with an updated version of the Moca panel. I utilized tutorials on HTML forms and JSP servlets to learn how to post data from the HTML text fields to the OpenMRS server. This data is stored as a new Observation within the current Encounter for this particular patient in OpenMRS.

JSP Tutorial

http://www.jsptut.com/Forms.jsp

Invoking Servlets and JSP files within HTML forms

http://publib.boulder.ibm.com/infocenter/wasinfo/v4r0/index.jsp?topic=/com.ibm.websphere.v35.doc/wass_content/040204040102.html

Handling Form Data

http://www.apl.jhu.edu/~hall/java/Servlet-Tutorial/Servlet-Tutorial-Form-Data.html

More details on how this panel integrates with the media viewer will come in my next post.

I just finished integrating image manipulation sliders into the media viewer so that you can change brightness, contrast, and sharpness of the image. Also zooming and panning works nicely too.

This is a brief summary of the issues I ran into in developing this feature and what references I consulted to solve them.

In order for the image to be displayed and manipulated in the Flash application, it had to be retrieved from OpenMRS. I created an Encounter Servlet that would HTTP get the images from OpenMRS for a given unique encounter ID. The media viewer module page in OpenMRS loads the viewer by sending the Flash SWF file the ID number of the encounter it wants to be displayed. Then Flash does HTTP GET to the Encounter servlet to retrieve all images, creates internal classes to hold the images and then displays them. In order to load an image from a URL, I used the solution posted here.

Three Ways to Load an Image in Flex

http://ntt.cc/2008/03/09/tips-three-ways-to-load-an-image-file-in-flex.html

When I mentioned that I created internal classes, I essentially created an encounter class in Actionscript with generic file classes. A generic file class would hold metadata and the actual image/audio/video object. In order to build instances of these classes with the correct information, the Encounter Servlet on OpenMRS would send XML summarizing the encounter information and the obs ID numbers of the images/audio/video files in OpenMRS. I had trouble with XML parsing and being able to retrieve the fields, but solved it using the example provided here. The key part was realizing that you could access data in a tag heading using the @ symbol: e = new Encounter(xmlStr.@patientFirst,xmlStr.@patientLast,xmlStr.@patientID);

Adobe Flex Documentation on XML Class

http://livedocs.adobe.com/flex/2/langref/XML.html

I researched many sites to find a good image processing library for image manipulation in this viewer. There were some with licensing issues, but in general there were good options with just using the built-in libraries. Hence, I decided to go with these libraries, but if we find that the resolution isn’t good enough for medical diagnosis, replacing the zoom, brightness, contrast, and sharpness methods with another library would be pretty straightforward since the code is modular.

Here are the resources that I used to develop the OpenMRS media viewer.

Flex Interface Guide (FIG): The Pan | Zoom Component from the Adobe Developer Connection

http://www.adobe.com/devnet/flex/samples/fig_panzoom/

Image Manipulation in Flex by Andrew Alderson

http://www.insideria.com/2008/03/image-manipulation-in-flex.html

BitmapUtil for Changing Image Contrast and Brightness in Actionscript by Stan Reshetnyk

http://flashcrafter.org.ua/content/bitmaputil-changing-image-contrast-brightness-actionscript

Okay! Below are the screenshots of the media viewer Flash application, which is embedded in the OpenMRS media viewer module.

What you see when the application launches. You can see the toolbar at the bottom with sliders where you can zoom, change sharpness, change contrast, and change brightness of the image. You can also zoom or pan by using the mouse directly over the image like in Google Maps (use the mouse scroll button to zoom in/out).

You can also use the options menu to hide the bottom toolbar to allow more screen space for the image. Even without the toolbar you can still zoom/pan with the mouse.

Zooming way in to the upper part of the chest.

This is an example of teledermatology and how you can play with the contrast slider. Original picture:

After changing the contrast:

Example of teleradiology and you can manipulate brightness. Original picture:

Making it darker. The original of this particular image was already pretty good, but if it was too bright, this would be useful!

Making it brighter.

Example of opthamology and manipulating sharpness level. Original picture: (From http://www.mymedicalcareer.com.au/specialties/ophthalmology/)

After picture. Looks sharp right?!

Coming up: Proper thumbnail downscaling for the preview strip in the bottom toolbar, Moca panel on the left for patient symptoms to go along with the image

Sorry, this is an overdue update! I left for a six-week trip to the Philippines to help deploy Moca in the Batanes province on a teleradiology project. It was the MOST amazing experience of my life. I learned so much by speaking to doctors, nurses, midwives, healthcare workers, and other locals on the ground. I was shocked and frustrated by the obstacles in the healthcare system that left millions without health insurance coverage and many more without proper access to quality care.

I will blog more about the trip in the coming posts, but I wanted to write about the extremely valuable input I received on the OpenMRS image viewer from the doctors and health professionals who I worked with. Below is a summary of the feedback I received when doing usability tests on them. I tried to gather research on the specific workflow that the media viewer would be used in so that I could tailor the system to integrate well with that workflow.

Radiologist

Usability test done on radiology oncologist (for radiation therapy for cancer patients) at Philippine General Hospital (PGH). He is a consultant with the radiology department at PGH and teaches residents. He voluntarily answers all teleradiology referrals for the National Telehealth Center by email.

Existing Workflow

•    Uses Mac image viewer called “Preview”
•    Uses Preview to modify darkness/brightness, white balance
•    “Eventually we want it like this (the OpenMRS image viewer). What we do now is difficult because I open the email, read it, etc..”
•    Receive only a few films by email, 10 per week
•    Willing to read up to 100 plates per day
•    Problems w/ x-rays (patient is breathing), radiology technician not properly trained
•    Send notification to cell phone to check the web
•    Will check email when get home, 1x per day, sometimes 2x
•    Each takes on average 1 minute to diagnose, chest x-ray 2 mins, max 5 mins
•    Slow internet at hospital
•    Doing telehealth referrals for pro bono now
•    1 patient per email so don’t mix up names with images
•    Takes 1 min to download the image, 2-3 minutes to diagnose

Take proper chest x-ray image (to retrain radiology technician)

•    Angulated by 15 degrees
•    See apex
•    Top part of chest should take up 50% of the x-ray
•    Female patient should remove bra

Features Needed

•    Zoom magnification
•    Vary contrast and brightness
•    Just save original image (not modified version)
•    Panel of information: symptoms, clinical diagnosis, clinical history
•    View past x-rays (2 side by side, previous and latest)
•    Access to older films (zoom and brightness capabilities on those images too)
•    Radiologists need box for “findings/interpretation” and for “notes”
•    Clinicians should be able to write diagnosis, treatment, notes (some fields can be blank)
•    No drop down box for diagnosis, no fixed reading (except for “normal chest”)
•    No cancel button needed, will read it then
•    If not sure of diagnosis, will leave it blank or refer to someone else  put case on “hold”, assign it to another doctor, allow both doctors to be on the case
•    Output an official report, type report next to image, then go directly to an editable version of the official report (auto populate the official report and then make it editable, preview + edit report)
•    Don’t need to annotate image, maybe a circle/arrow/pointer if clinician wants to see lesion (dermatology cases may require annotations)
•    List of on-call doctors, typically on-call all day from 7am-11pm, diff person per day, a doctor is on-call 2-3 times per week
•    Could charge 30-50 pesos per x-ray plate reading
•    If bad image, disappear from queue until need it again, don’t need the bad pics
•    Personal digital signature

Nurse

Difficulties

•    Double click to annotate

Workflow

•    Clinician needs to incorporate the radiologist reading with other data from patient (ECGs, other labs)
•    Once sure about interpretation, don’t need to go back and save, only compare, won’t return to edit it
•    Clinician  rad tech  radiologist  clinician
•    Time diagnosis 48 hrs
•    Use Windows default image viewer

Features Needed

•    Make the image DICOM compliant (Open binary file, first part is text (key value pairs in header), parse into fields, latter part is image
•    Embed in file the doctor who interpreted the data
•    Click on annotation to edit again
•    Zoom (enlarge it so one lung is screen width, 3-5 MP)
•    Don’t need to rotate it
•    Label it as “user” or “patient”
•    Add doctor’s name as the one interpreting the data
•    Annotation box can be fixed to a corner of the screen
•    Don’t need other info from medical record of patient besides the summary
•    Compare with previous x-ray films, click to see previous plates, side by side comparison
•    If not sure, need to refer to another doctor (residents may be unsure about diagnosis)
•    Clinician needs diagnosis, treatment, other notes box
•    Clinician may not need to annotate image
•    Can have another system for clinician, pull out radiology image for same visit of patient
•    Change brightness, contrast, sometimes can see shadows
•    Date is fine, don’t need timestamp
•    Bad image quality -> comment why bad image, want another view
•    Retake shot (i.e. top of chest)
•    Save original ones with new images
•    Drag image around, zoom to click, scroll image like google maps

Doctors in Batanes

•    Need annotations for clinicians and radiologists
•    Blank spot for clinical diagnosis
•    Blank spot for radiologist (i.e. there were no densities, diaphragm is intact, heart is not enlarged)

Doctor in Manila

•    Spell checking for the boxes
•    Findings, diagnosis boxes
•    Make sure it’s embedded in the workflow (see surgery system by National Telehealth Center)
•    Separate screen for diagnosis of patient, not in the image viewer