this post was submitted on 03 Nov 2024
390 points (98.0% liked)

Open Source

31365 readers
163 users here now

All about open source! Feel free to ask questions, and share news, and interesting stuff!

Useful Links

Rules

Related Communities

Community icon from opensource.org, but we are not affiliated with them.

founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] Cr4yfish@lemmy.world 8 points 2 weeks ago

I use Gemini, which supports PDF File uploads, combined with structured outputs to generate Course Sections, Levels & Question JSON.

When you upload a PDF, it first gets uploaded to a S3 Database directly from the Browser, which then sends the Filename and other data to the Server. The Server then downloads that Document from the S3 and sends it to Gemini, which then streams JSON back to the Browser. After that, the PDF is permanently deleted from the S3.

Data Privacy wise, I wouldn't upload anything sensitive since idk what Google does with PDFs uploaded to Gemini.

The Prompts are in English, so the output language is English as well. However, I actually only tested it with German Lecture PDFs myself.

So, yes, it probably works with any language that Gemini supports.

Here is the Source Code for the core function for this feature:

export async function createLevelFromDocument(
    { docName, apiKey, numLevels, courseSectionTitle, courseSectionDescription }: 
    { docName: string, apiKey: string, numLevels: number, courseSectionTitle: string, courseSectionDescription: string }) 
    {
    
    const hasCourseSection = courseSectionTitle.length > 0 && courseSectionDescription.length > 0;

    // Step 1: Download the PDF and get a buffer from it
    const blob = await downloadObject({ filename: docName, path: "/", bucketName: "documents" });
    const arrayBuffer = await blob.arrayBuffer();
    
    // Step 2: call the model and pass the PDF
    //const openai = createOpenAI({ apiKey: apiKey });
    const gooogle = createGoogleGenerativeAI({ apiKey: apiKey });

    const courseSectionsPrompt = createLevelPrompt({ hasCourseSection, title: courseSectionTitle, description: courseSectionDescription });
    
    const isPDF = docName.endsWith(".pdf");

    const content: UserContent = [];

    if(isPDF) {
        content.push(pdfUserMessage(numLevels, courseSectionsPrompt) as any);
        content.push(pdfAttatchment(arrayBuffer) as any);
    } else {
        const html = await blob.text();
        content.push(htmlUserMessage(numLevels, courseSectionsPrompt, html) as any);
    }

    const result = await streamObject({ 
        model: gooogle("gemini-1.5-flash"),
        schema: multipleLevelSchema,
        messages: [
            {
                role: "user",
                content: content
            }
        ]
    })
    

    return result;
}