Falcon3-7B-Instruct: Optimized for Mobile Deployment

State-of-the-art large language model useful on a variety of language understanding and generation tasks

Falcon3 family of Open Foundation Models is a set of pretrained and instruct LLMs ranging from 1B to 10B.

This model is an implementation of Falcon3-7B-Instruct found here.

This repository provides scripts to run Falcon3-7B-Instruct on Qualcomm® devices. More details on model performance across various devices, can be found here.

Model Details

• Model Type: Model_use_case.text_generation
• Model Stats:
  • Input sequence length for Prompt Processor: 128
  • Context length: 4096
  • Precision: w4a16 + w8a16 (few layers)
  • Num of key-value heads: 4
  • Model-1 (Prompt Processor): PromptProcessor
  • Prompt processor input: 128 tokens + position embeddings + attention mask + KV cache inputs
  • Prompt processor output: 128 output tokens + KV cache outputs
  • Model-2 (Token Generator): TokenGenerator
  • Token generator input: 1 input token + position embeddings + attention mask + KV cache inputs
  • Token generator output: 1 output token + KV cache outputs
  • Use: Initiate conversation with prompt-processor and then token generator for subsequent iterations.
  • Supported languages: English, French, Spanish, Portuguese.
  • Minimum QNN SDK version required: 2.28.2
  • TTFT: Time To First Token is the time it takes to generate the first response token. This is expressed as a range because it varies based on the length of the prompt. The lower bound is for a short prompt (up to 128 tokens, i.e., one iteration of the prompt processor) and the upper bound is for a prompt using the full context length (4096 tokens).
  • Response Rate: Rate of response generation after the first response token.

| Model | Precision | Device | Chipset | Target Runtime | Response Rate (tokens per second) | Time To First Token (range, seconds) |---|---|---|---|---|---| | Falcon3-7B-Instruct | w4a16 | Snapdragon 8 Elite Gen 5 QRD | Snapdragon® 8 Elite Gen5 Mobile | GENIE | 15.8303 | 0.10903 - 3.488966 | -- | Use Export Script | | Falcon3-7B-Instruct | w4a16 | Snapdragon 8 Elite QRD | Snapdragon® 8 Elite Mobile | GENIE | 14.02985 | 0.1265205 - 4.048656 | -- | Use Export Script | | Falcon3-7B-Instruct | w4a16 | Snapdragon X Elite CRD | Snapdragon® X Elite | GENIE | 9.96829 | 0.1973798 - 6.3161536 | -- | Use Export Script |

Deploying Falcon3-7B-Instruct on-device

Please follow the LLM on-device deployment tutorial.

Installation

Install the package via pip:

# NOTE: 3.10 <= PYTHON_VERSION < 3.14 is supported.
pip install "qai-hub-models[falcon-v3-7b-instruct]"

Configure Qualcomm® AI Hub Workbench to run this model on a cloud-hosted device

Sign-in to Qualcomm® AI Hub Workbench with your Qualcomm® ID. Once signed in navigate to Account -> Settings -> API Token.

With this API token, you can configure your client to run models on the cloud hosted devices.

qai-hub configure --api_token API_TOKEN

Navigate to docs for more information.

Demo off target

The package contains a simple end-to-end demo that downloads pre-trained weights and runs this model on a sample input.

python -m qai_hub_models.models.falcon_v3_7b_instruct.demo

The above demo runs a reference implementation of pre-processing, model inference, and post processing.

NOTE: If you want running in a Jupyter Notebook or Google Colab like environment, please add the following to your cell (instead of the above).

%run -m qai_hub_models.models.falcon_v3_7b_instruct.demo

Run model on a cloud-hosted device

In addition to the demo, you can also run the model on a cloud-hosted Qualcomm® device. This script does the following:

• Performance check on-device on a cloud-hosted device
• Downloads compiled assets that can be deployed on-device for Android.
• Accuracy check between PyTorch and on-device outputs.

python -m qai_hub_models.models.falcon_v3_7b_instruct.export

Deploying compiled model to Android

The models can be deployed using multiple runtimes:

• TensorFlow Lite (.tflite export): This tutorial provides a guide to deploy the .tflite model in an Android application.

• QNN (.so export ): This sample app provides instructions on how to use the .so shared library in an Android application.

View on Qualcomm® AI Hub

Get more details on Falcon3-7B-Instruct's performance across various devices here. Explore all available models on Qualcomm® AI Hub

License

• The license for the original implementation of Falcon3-7B-Instruct can be found here.
• The license for the compiled assets for on-device deployment can be found here

References

• Source Model Implementation

Community

• Join our AI Hub Slack community to collaborate, post questions and learn more about on-device AI.
• For questions or feedback please reach out to us.