Project

# Title Team Members TA Documents Sponsor
23 Portable RAW Reconstruction Accelerator for Legacy CCD Imaging
 Guyan Wang
Yuhong Chen
 other1.docx
# **RFA: Portable RAW Reconstruction Accelerator for Legacy CCD Imaging**

Group Member: Guyan Wang, Yuhong Chen

## **1\. Problem Statement**

**The "Glass-Silicon Gap":** Many legacy digital cameras (circa 2000-2010) are equipped with premium optics (Leica, Zeiss, high-grade Nikon/Canon glass) that outresolve their internal processing pipelines. While the optical pathway is high-fidelity, the final image quality is bottlenecked by:

- **Obsolete Signal Chains:** Early-stage Analogue-to-Digital Converters (ADCs) and readout circuits introduce significant read noise and pattern noise.
- **Destructive Processing:** In-camera JPEGs destroy dynamic range and detail. Even legacy RAW files are often processed with rudimentary demosaicing algorithms that fail to distinguish high-frequency texture from sensor noise.
- **Usability Void:** Users seeking the unique "CCD look" are forced to rely on cumbersome desktop post-processing workflows (e.g., Lightroom, Topaz), preventing a portable, shoot-to-share experience.

## **2\. Solution Overview**

**The "Digital Back" External Accelerator:** We propose a standalone, handheld hardware device-a "smart reconstruction box"-that interfaces physically with legacy CCD cameras. Instead of relying on the camera's internal image processor, this device ingests the raw sensor data (CCD RAW) and applies a hybrid reconstruction pipeline.

The core innovation is a **Hardware-Oriented Hybrid Pipeline**:

- **Classical Signal Processing:** Handles deterministic error correction (black level subtraction, gain normalization, hot pixel mapping).
- **Learned Estimator (AI):** A lightweight Convolutional Neural Network (CNN) or Vision Transformer model optimized for microcontroller inference (TinyML). This model does not "hallucinate" new details but acts as a probabilistic estimator to separate signal from stochastic noise based on the physics of CCD sensor characteristics.

The device will feature a touchscreen interface for file selection and "film simulation" style filter application, targeting an output quality perceptually comparable to a modern full-frame sensor (e.g., Sony A7 III) in terms of dynamic range recovery and noise floor.

## **3\. Solution Components**

### **Component A: The Compute Core (Embedded Host)**

- **MCU:** STMicroelectronics **STM32H7 Series** (e.g., STM32H747/H757).
- _Rationale:_ Dual-core architecture (Cortex-M7 + M4) allows separation of UI logic and heavy DSP operations. The Chrom-ART Accelerator helps with display handling, while the high clock speed supports the computationally intensive reconstruction algorithms.
- **Memory:** External SDRAM/HyperRAM expansion (essential for buffering full-resolution RAW files, e.g., 10MP-24MP) and high-speed QSPI Flash for AI model weight storage.

### **Component B: Connectivity & Data Ingestion Interface**

- **Physical I/O:** USB OTG (On-The-Go) Host port.
- _Function:_ The device acts as a USB Host, mounting the camera (or the camera's card reader) as a Mass Storage Device to pull RAW files (.CR2, .NEF, .RAF, .DNG).
- **Storage:** On-board MicroSD card slot for saving processed/reconstructed JPEGs or TIFFs.

### **Component C: Hybrid Reconstruction Algorithm**

- **Stage 1 (DSP):** Linearization, dark frame subtraction (optional calibration), and white balance gain application.
- **Stage 2 (NPU/AI):** A quantization-aware trained model (likely TFLite for Microcontrollers or STM32-AI) trained specifically on _noisy CCD -to- clean CMOS_ image pairs.
- _Task:_ Joint Demosaicing and Denoising (JDD).
- **Stage 3 (Color):** Application of specific "Film Looks" (LUTs) selected by the user via the UI.

### **Component D: Human-Machine Interface (HMI)**

- **Display:** 2.8" to 3.5" Capacitive Touchscreen (SPI or MIPI DSI interface).
- **GUI Stack:** TouchGFX or LVGL.
- _Workflow:_ User plugs in camera -> Device scans for RAWs -> User selects thumbnails -> User chooses "Filter/Profile" -> Device processes and saves to SD card.

## **4\. Criterion for Success**

To be considered successful, the prototype must meet the following benchmarks:

- **Quality Parity:** The output image, when blind-tested against the same scene shot on a modern CMOS sensor (Sony A7 III class), must show statistically insignificant differences in perceived noise at ISO 400-800 equivalent.
- **Edge Preservation:** The AI reconstruction must demonstrate a reduction in color moiré and false-color artifacts compared to standard bilinear demosaicing, without "smoothing" genuine texture (measured via MTF charts).
- **Latency:** Total processing time for a 10-megapixel RAW file must be under **15 seconds** on the STM32 hardware.
- **Universal RAW Support:** Successful parsing and decoding of at least two major legacy formats (e.g., Nikon .NEF from D200 era and Canon .CR2 from 5D Classic era).

## **5\. Alternatives**

- **Desktop Post-Processing (Software Only):**
- _Pros:_ Infinite computing power, established tools (DxO PureRAW), highly customized.
- _Cons:_ Destroys the portability of the photography experience; cannot be done "in the field." Need to be proficient with parameters inside the software, which requires self-training and tutoring (not user-friendly).
- **Smartphone App (via USB-C dongle):**
- _Pros:_ Powerful processors (Snapdragon/A-Series), high-res screens, easy to use.
- _Cons:_ Lack of low-level control over USB mass storage protocols for obscure legacy cameras; high friction in file management; operating system overhead prevents bare-metal optimization of the signal pipeline; unique algorithms may not be suitable for legacy cameras.
- **FPGA Implementation (Zynq/Cyclone):**
- _Pros:_ Parallel processing could make reconstruction instant.
- _Cons:_ Significantly higher complexity, cost, and power consumption compared to an STM32 implementation; higher barrier to entry for a "mini project."

RFA: Any-Screen to Touch-Screen Device

Ganesh Arunachalam, Sakhi Yunalfian

Featured Project

# Any-Screen to Touch-Screen Device

Team Members:

\- Sakhi Yunalfian (sfy2)

\- Muthu Arunachalam (muthuga2)

\- Zhengjie Fan (zfan11)

# Problem

While touchscreens are becoming increasingly popular, not all screens come equipped with touch capabilities. Upgrading or replacing non-touch displays with touch-enabled ones can be costly and impractical. Users need an affordable and portable solution that can turn any screen into a fully functional touchscreen.

# Solution

The any-screen-to-touch-screen device uses four ultra-wideband sensors attached to the four corners of a screen to detect the position of a specially designed pen or hand wearable. Ultrawideband (UWB) is a positioning technology that is lower-cost than Lidar/Camera yet more accurate than Bluetooth/Wifi/RFID. Since UWB is highly accurate we will use these sensors to track the location of a UWB antenna (placed in the pen). In addition to the UWB tag, the pen will also feature a touch-sensitive tip to detect contact with the screen (along with a redundant button to simulate screen contact if the user prefers to not constantly make contact with the screen). The pen will also have a gyroscope and low profile buttons to track tilt data and offer customizable hotkeys/shortcuts. The pen and sensors communicate wirelessly with the microcontroller which converts the pen’s input data along with its location on the screen into touchscreen-like interactions.

# Solution Components

## Location Sensing Subsystem (Hardware)

This subsystem will employ Spark Microsystems SR1010 digitally programmable ultra-wideband wireless transceiver. The transceiver will be housed in a enclosure that can be attached to the corners of a screen or monitor. Each sensor unit will also need a bluetooth module in order to communicate with the microcontroller.

## Signal Processing Subsystem (Hardware and Software)

A microcontroller, specifically the STM32F4 series microcontroller (STM32F407 or STM32F429). Real-time sensor data processing takes away a considerable amount of computing power. The STM32F4 series contain DSP instructions that allow a smoother way to perform raw data processing and noise reduction. This subsystem will allow us to perform triangulation to accurately estimate the location on the screen, smooth real-time data processing, latency minimization, sensitivity, and noise reduction.

A bluetooth module, in order for the sensor to send its raw data to the microcontroller. We are planning to make the communication between the sensors and the pen to the microcontroller to be wireless. One bluetooth module we are considering is the HC05 bluetooth module.

The microcontroller itself will be wired to the relevant computer system via USB 2.0 for data transfer of touchscreen interactions.

## Pen/Hand Wearable Subsystem (Hardware)

The pen subsystem will employ a simple spring switch as a pen tip to detect pen to screen contact. We will also use a Sparkfun DEV-08776 Lilypad button to simulate a press/pen-to-screen contact for redundancy and if the user wishes to control the pen without contact to the screen. The pen will also contain several low profile buttons and a STMicroelectronics LSM6DSO32TR gyroscope/accelerator sensor to provide further customizable pen functionality and potentially aid in motion tracking calculations. The pen will contain a Taoglas UWC.01 ultra-wideband tag to allow detection by the location sensing subsystem and a bluetooth module to allow communication with the microcontroller. The unit will need to be enclosed within a plastic or 3D printed housing.

## Touch Screen Emulation Subsystem (Software)

A microcontroller with embedded HID device functionalities in order to control mouse cursors of a specific device connected to it. We are planning to utilize the STM32F4 series microcontroller with built in USB HID libraries to help emulating the touch screen effects. We will also include a simple GUI to allow the user to customize the shortcuts mapped to the pen buttons and specify optional parameters like screen resolution, screen curve, etc.

## Power Subsystem (Hardware)

The power subsystem is not localized in one area since our solution consists of multiple wireless devices, however we specify all power requirements and solutions here for organization purposes.

For the wireless sensors in our location sensing subsystem, we plan on using battery power. Given the UWB transceiver has ultra-low power consumption and an internal DC-DC converter, it makes sense to power each sensor unit with a small 3.3V 650mAh rechargeable battery (potential option: [https://a.co/d/acFLsSu](https://a.co/d/acFLsSu)). We will include recharging capability and micro usb recharging port.

For our pen, we plan on using battery power too. The gyroscope module, UWB antenna, and bluetooth module all have low-power consumption so we plan on using the same rechargeable battery system as specified above.

The microcontroller will be wired via USB 2.0 directly to the computer subsystem in order to transmit mouse data/touchscreen interaction and will receive 5V 0.9A power supply through this connection.

# Criterion For Success

## Hardware

The UWB sensor system is able to track the pens location on the screen.

The pen is able to detect clicks, screen contact, and tilt.

The microcontroller is able to take input from the wireless pen and the wireless sensors.

Each battery-powered unit is successfully powered and able to be charged.

## Software

The pen’s input and sensor location data can be converted to mouse clicks and presses.

The pen’s buttons can be mapped to customizable shortcuts/hotkeys.

## Accuracy and Responsiveness

Touch detection and location accuracy is the most crucial criteria for our project’s success. We expect our device to have a 95% touch detection precision. In order to correctly control embedded HID protocols of a device, the data sent and processed by the microcontroller to the device has to have a low error threshold when comparing cursor movements with wearable location.

Touch recognition and responsiveness is the next most important thing. We want our system, by a certain distance threshold, able to detect the device with a relatively low margin of error of about 1% or less. More specifically, this criteria for success is the conclusion to see if our communication network protocol between the sensors, USB HID peripherals, and the microcontroller are able to efficiently transfer data in real-time for the device to interpret these data in a form of cursor location updates, scrolls, clicks, and more.

Latency and lags should have a time interval of less than 60 millisecond. This will be judged based on the DSP pipeline formed in the STM32F4 microcontroller.

## Reliability and Simplicity

We want our device to be easily usable for the users. It should be intuitive and straightforward to start the device and utilize its functionalities.

We want our device to also be durable in the sense of low chances of battery failures, mechanical failures, and systematic degradations.

## Integration and Compatibility

We want our device to be able to be integrated with any type of screens of different architectural measurements and operating systems.

Project Videos