Smart Assistant with edge AI computing - Summer Internship at Himax Imaging in Irvine (Ongoing)

Introduction

1. to develop and improve the computer vision AI model residing within a small chip (or more formally, an AI module product ISM-028) designed by the other team of our company, and also

2. to build a laptop application for demonstration of the capability and applicability of our chip.

Holding two master degrees of electrical engineering and computer science respectively, this intern opportunity really fits my background and interest. Especially with my previous project Hand-Writing Robot,

AI-Powered Driver Companion Mobile App, Deep-Learning based Natural Language Processing research, the experience in software quick prototyping, and deep learning research equips me with a lot of skills and knowledge that I can tap into for this intern.

Background

1. to develop advanced AI-powered features provided by the AI chip ISM-028, our company designed

2. to optimize models in accuracy and speed

3. to demonstrate the usability and potential of ISM-028.

What is an AI PC?

Endpoint AI vs Edge AI vs Cloud AI

Always-On-Sensing (AOV) and Always-on-Vision (AONV)

Goals

Model Development for More Downstream Computer Vision Tasks

Model Optimization

Windows Desktop Application Development

Framework and Tool Decision For Windows Desktop Application Development

System Diagram

UI Design

Environment Setup for React-Native

Using CLI (Failed)

$ Set-ExecutionPolicy Unrestricted -Scope Process -Force
$ iex (New-Object System.Net.WebClient).DownloadString('https://aka.ms/rnw-vs2022-deps.ps1');

$ npx --yes react-native@latest init wise --version "latest"
$ cd wise
wise\ $ npx --yes react-native-windows-init --overwrite

× Failed to deploy: ERROR: ReflectionTypeLoadException: Unable to load one or more of the requested types. Retrieve the LoaderExceptions property for more information.                                                                

Command failed. Re-run the command with --logging for more information.
 × Deploying C:/Users/550030/wise/windows/x64/Debug/wise/wise.build.appxrecipe: ERROR: ERROR: ReflectionTypeLoadE...
 
ERROR: Exceptions from the reflection loader:
ERROR: FileLoadException: Could not load file or assembly 'NuGet.VisualStudio.Contracts, Version=17.10.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

• Failed to deploy. Package NuGet.VisualStudio.Contracts was missing.
• The package NuGet.VisualStudio.Contracts contains RPC contracts for NuGet’s Visual Studio Service Broker extensibility APIs. These APIs are designed to be usable with async code and are available in this package using Visual Studio’s IServiceBroker.
• dotnet list windows\wise.sln package also does not work 😟
The error shows that SolutionDir environment variable is empty and the node_modules\react-native-windows\package.json could not be found. ⇒ why is it empty? set the variable as a workaround?

With Visual Studio Code (Failed 😟)

• install extension

• create a file in .vscode\launch.json with the configuration for Debug

• run Debug

With Visual Studio (It works!)

• npx react-native autolink-windows

• build, deploy and run on local machine or just debug

Environment Setup for Electron with React

• vite-plugin-electron

Setup for React and Electron with TypeScript

$ yarn add -D vite

$ yarn create vite
√ Select a framework: » Others
√ Select a variant: » create-electron-vite ↗
√ Project template: » React
# this will come with TypeScript, React, and Electron, but it has some issues installing electron
# using this script. 
[1/2] ⢀ esbuild
error C:\Users\550030\wise-electron\node_modules\electron: Command failed.
Exit code: 1
Command: node install.js
Arguments:
Directory: C:\Users\550030\wise-electron\node_modules\electron
Output:
RequestError: unable to verify the first certificate

# to solve this, install electron manually by `yarn add --dev electron`

$ yarn install

$ yarn dev

Setup for Tailwind CSS

$ yarn add --dev tailwindcss postcss autoprefixer

# tailwindcss init flags:
#       --esm                Initialize configuration file as ESM
#       --ts                 Initialize configuration file as TypeScript
#   -p, --postcss            Initialize a `postcss.config.js` file
#   -f, --full               Include the default values for all options in the generated configuration file
$ npx tailwindcss init -p
Created Tailwind CSS config file: tailwind.config.js
Created PostCSS config file: postcss.config.js

  content: [
    "./index.html",
    "./src/**/*.{js,ts,jsx,tsx}",
  ],

@tailwind base;
@tailwind components;
@tailwind utilities;

Use react-chatbot-kit

Chatbot/
	config.js
	MessageParser.jsx
	ActionProvider.jsx
	
	Chatbot.tsx
	Chatbot.css

import { createChatBotMessage } from 'react-chatbot-kit';

const config = {
  initialMessages: [createChatBotMessage(`Hello world`)],
};

export default config;

import React from 'react';

const ActionProvider = ({ createChatBotMessage, setState, children }) => {
  return (
    <div>
      {React.Children.map(children, (child) => {
        return React.cloneElement(child, {
          actions: {},
        });
      })}
    </div>
  );
};

export default ActionProvider;

import React from 'react';

const MessageParser = ({ children, actions }) => {
  const parse = (message) => {
    console.log(message);
  };

  return (
    <div>
      {React.Children.map(children, (child) => {
        return React.cloneElement(child, {
          parse: parse,
          actions: {},
        });
      })}
    </div>
  );
};

export default MessageParser;

import Chatbot from 'react-chatbot-kit'
import 'react-chatbot-kit/build/main.css'

import config from './config.js';
import MessageParser from './Messageparser.js';
import ActionProvider from './ActionProvider.js';

export const MyChatbot = () => {... 
  return (
    <div>
      <Chatbot
        config={config}
        messageParser={MessageParser}
        actionProvider={ActionProvider}
      />
    </div>
  );
	...
};

Chatbot Integration

The First Natural Command - Set Brightness

• Renderer process that runs within a browser that can not have access to the OS directly (like a virtual environment or a virtual world)

• Main process that runs on the operating system, just like back-end, which has access to the OS directly, and has controls over the browser that the renderer process runs within and the application lifetime

C++ Event Listener

• FT4222 Library

• Check EPII_ISP_TOOL

Django Backend

• django-rest-framework (DRF)

• django-filters

• drf-spectacular : for API documentation with Swagger/Redoc and client generation with OpenAPI

Client Generation by OpenAPI

(wise-backend) wise_backend/ $ python manage.py spectacular > schema.yml
wise-electron/ $ openapi-generator-cli generate -i ..\wise_backend\schema.yml -g typescript-fetch -o .\src\services

To Investigate

1. C/C++ addons for nodejs
1. rollup
2. node api
3. node.h
4. NAN

2. C# / .NET as Backend

3. .NET Core with TypeScript