Summary
Here is the summary of the improvements (continued work from W2 LongMemEval Inspect & KG Consolidation):
- Better System Prompt: Now IE module will try to capture more relations (including reasoning) from the context.
- Pure KG oriented Json output → Structured Json Output: Ensure the KG interaction module works for all output.
- Better Graph Visualisation
Compare Result
Visualisation
Ref to previous version: Result
The improved version:
IE Module Output Json
Ref to previous version: IE Module Output Json
The improved version:
{
"add_node": [
{
"node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"node_details": {
"label": "Digital Task Management App Query",
"timestamp": "2023/05/30T17:27:00Z",
"description": "User inquires about digital task management apps to help prioritize work and personal tasks, seeking a digital alternative to a physical planner."
}
},
{
"node_id": "/Knowledge/Concept/ProductivityTool/Planner",
"node_details": {
"label": "Planner",
"description": "A physical tool used for organizing tasks and appointments."
}
},
{
"node_id": "/Knowledge/SoftwareApplication/ProductivityTool/Todoist",
"node_details": {
"label": "Todoist",
"description": "A highly-rated app with a clean interface, allowing task creation, management, priority setting, and progress tracking. Features include labels, filters, and integrations."
}
},
{
"node_id": "/Knowledge/SoftwareApplication/ProductivityTool/Trello",
"node_details": {
"label": "Trello",
"description": "A visual project management tool using boards, lists, and cards to organize tasks, suitable for Kanban-style workflow management with a vast library of integrations."
}
},
{
"node_id": "/Knowledge/SoftwareApplication/ProductivityTool/Asana",
"node_details": {
"label": "Asana",
"description": "A robust task management platform for creating, assigning tasks, setting deadlines, and tracking progress, ideal for teams but also suitable for individual use."
}
},
{
"node_id": "/Knowledge/SoftwareApplication/ProductivityTool/RescueTime",
"node_details": {
"label": "RescueTime",
"description": "A time management app that tracks computer and mobile device usage, providing detailed productivity reports to identify areas for improvement."
}
},
{
"node_id": "/Knowledge/SoftwareApplication/ProductivityTool/Any.do",
"node_details": {
"label": "Any.do",
"description": "A user-friendly task management app with a simple, intuitive interface, offering features like task assignment, reminders, and collaboration tools."
}
},
{
"node_id": "/Knowledge/SoftwareApplication/ProductivityTool/Wunderlist",
"node_details": {
"label": "Wunderlist",
"description": "A popular, easy-to-use app for creating and managing to-do lists, featuring a clean design, reminders, and collaboration features."
}
},
{
"node_id": "/Knowledge/SoftwareApplication/ProductivityTool/ClickUp",
"node_details": {
"label": "ClickUp",
"description": "An all-in-one project management platform that includes task management, time tracking, and team collaboration features."
}
},
{
"node_id": "/Knowledge/SoftwareApplication/ProductivityTool/MicrosoftToDo",
"node_details": {
"label": "Microsoft To Do",
"description": "A simple, free task management app from Microsoft that integrates well with Outlook and other Microsoft services."
}
},
{
"node_id": "/Knowledge/SoftwareApplication/ProductivityTool/Evernote",
"node_details": {
"label": "Evernote",
"description": "A note-taking app that also offers task management features, such as creating to-do lists and setting reminders."
}
},
{
"node_id": "/Knowledge/SoftwareApplication/ProductivityTool/GoogleTasks",
"node_details": {
"label": "Google Tasks",
"description": "A basic, free task management app from Google that integrates well with Google Calendar and Gmail."
}
},
{
"node_id": "/Knowledge/Concept/ProductivityTool/DigitalTaskManagementSystem",
"node_details": {
"label": "Digital Task Management System",
"description": "A digital tool or software application designed to help users organize, prioritize, and track their tasks and projects."
}
},
{
"node_id": "/Knowledge/Concept/EvaluationCriteria/EaseOfUse",
"node_details": {
"label": "Ease of use",
"description": "How intuitive is the app? Can you easily create and manage tasks?"
}
},
{
"node_id": "/Knowledge/Concept/EvaluationCriteria/Features",
"node_details": {
"label": "Features",
"description": "Do you need features like collaboration, time tracking, or integrations with other apps?"
}
},
{
"node_id": "/Knowledge/Concept/EvaluationCriteria/Customization",
"node_details": {
"label": "Customization",
"description": "Can you tailor the app to fit your workflow and prioritization needs?"
}
},
{
"node_id": "/Knowledge/Concept/EvaluationCriteria/Accessibility",
"node_details": {
"label": "Accessibility",
"description": "Is the app available on multiple platforms (web, mobile, desktop)?"
}
},
{
"node_id": "/Knowledge/Concept/EvaluationCriteria/Cost",
"node_details": {
"label": "Cost",
"description": "Are you willing to pay for premium features, or do you prefer a free app?"
}
}
],
"update_node": [
{
"node_id": "/User/Status/CurrentFocus",
"node_details": {
"label": "Organizing life better",
"description": "User is focused on improving personal organization and productivity, specifically through task management."
}
},
{
"node_id": "/User/Status/ImmediateNeeds",
"node_details": {
"label": "Task management app recommendations",
"description": "User needs recommendations for digital task management apps that can help prioritize work and personal tasks."
}
},
{
"node_id": "/User/Status/InteractionState",
"node_details": {
"label": "Seeking Recommendations",
"description": "User is actively seeking recommendations for productivity tools."
}
},
{
"node_id": "/User/Profile/ToolsUsedHistory",
"node_details": {
"label": "Planner",
"description": "User has been using a physical planner for task management."
}
},
{
"node_id": "/Assistant/Status/CurrentInteractionContext",
"node_details": {
"label": "Providing Task Management App Recommendations",
"description": "Assistant is providing recommendations for digital task management apps based on user's need for better organization and task prioritization."
}
}
],
"add_relation": [
{
"source_node_id": "/User/RelatedEvents",
"target_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"relation_details": {
"label": "INITIATED"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/User/Status/CurrentFocus",
"relation_details": {
"label": "INDICATES_USER_FOCUS"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/User/Status/ImmediateNeeds",
"relation_details": {
"label": "SEEKS"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/Concept/ProductivityTool/Planner",
"relation_details": {
"label": "REPLACES_PREVIOUS_TOOL"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/Concept/ProductivityTool/DigitalTaskManagementSystem",
"relation_details": {
"label": "TARGETS_SOLUTION_TYPE"
}
},
{
"source_node_id": "/Assistant/RelatedEvents",
"target_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"relation_details": {
"label": "RESPONDED_TO"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/SoftwareApplication/ProductivityTool/Todoist",
"relation_details": {
"label": "RECOMMENDED_APP"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/SoftwareApplication/ProductivityTool/Trello",
"relation_details": {
"label": "RECOMMENDED_APP"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/SoftwareApplication/ProductivityTool/Asana",
"relation_details": {
"label": "RECOMMENDED_APP"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/SoftwareApplication/ProductivityTool/RescueTime",
"relation_details": {
"label": "RECOMMENDED_APP"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/SoftwareApplication/ProductivityTool/Any.do",
"relation_details": {
"label": "RECOMMENDED_APP"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/SoftwareApplication/ProductivityTool/Wunderlist",
"relation_details": {
"label": "RECOMMENDED_APP"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/SoftwareApplication/ProductivityTool/ClickUp",
"relation_details": {
"label": "RECOMMENDED_APP"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/SoftwareApplication/ProductivityTool/MicrosoftToDo",
"relation_details": {
"label": "RECOMMENDED_APP"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/SoftwareApplication/ProductivityTool/Evernote",
"relation_details": {
"label": "RECOMMENDED_APP"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/SoftwareApplication/ProductivityTool/GoogleTasks",
"relation_details": {
"label": "RECOMMENDED_APP"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/Concept/EvaluationCriteria/EaseOfUse",
"relation_details": {
"label": "SUGGESTED_EVALUATION_CRITERIA"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/Concept/EvaluationCriteria/Features",
"relation_details": {
"label": "SUGGESTED_EVALUATION_CRITERIA"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/Concept/EvaluationCriteria/Customization",
"relation_details": {
"label": "SUGGESTED_EVALUATION_CRITERIA"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/Concept/EvaluationCriteria/Accessibility",
"relation_details": {
"label": "SUGGESTED_EVALUATION_CRITERIA"
}
},
{
"source_node_id": "/Events/PersonalOrganization/DigitalTaskManagementQuery_20230530",
"target_node_id": "/Knowledge/Concept/EvaluationCriteria/Cost",
"relation_details": {
"label": "SUGGESTED_EVALUATION_CRITERIA"
}
}
]
}System Prompt for IE Module
Ref to previous version: System Prompt for IE Module
The improved version:
You are an expert Knowledge Graph Engineer AI. Your task is to analyze a user-assistant dialogue segment and generate a list of explicit operations to build and update a knowledge graph (KG). **YOU MUST parse EVERY word from the dialogue and create or update nodes and relations in the KG. (Which means we must be able to use the parsed KG to rebuild original dialogue.)** The KG has a specific 4-root-node structure.
**Knowledge Graph (KG) Structure Overview:**
1. **`/User`**: Singular root node for the human user.
* Fixed Sub-nodes: `/User/Profile`, `/User/Status`, `/User/RelatedEvents`.
* `/User/Status` has further fixed sub-nodes like `/User/Status/CurrentFocus`, `/User/Status/CurrentMood`, `/User/Status/ImmediateNeeds`, `/User/Status/InteractionState`.
* `/User/Profile` has sub-nodes like `/User/Profile/Interests`, `/User/Profile/ToolsUsedHistory`, `/User/Profile/StatedGoals`.
2. **`/Assistant`**: Singular root node for the AI assistant.
* Fixed Sub-nodes: `/Assistant/Profile`, `/Assistant/Persona`, `/Assistant/Status`, `/Assistant/RelatedEvents`.
* `/Assistant/Status` has sub-nodes like `/Assistant/Status/CurrentInteractionContext`.
3. **`/Events`**: Root collection for interaction events.
* Structure: `/Events/[Topic]/[EventNodeID]`
* `[Topic]` examples: "PersonalOrganization", "AcademicQuery", "TechnicalSupport", "HealthAndWellness", "SocialChat", "FeedbackSession", "GeneralQuery". (Your consolidation logic will use a predefined list or allow dynamic creation based on LLM suggestion).
4. **`/Knowledge`**: Root collection for general knowledge entities.
* Structure: `/Knowledge/[EntityType]/[EntityCategory]/[EntityNameAsID]`
* `[EntityType]` examples: `SoftwareApplication`, `AnalogTool`, `Concept`, `Topic`, `Person`, `Organization`.
* `[EntityCategory]` can be used for further classification (if not applicable, use the same name as EntityType). For example, `/Knowledge/Organization/Company`, `/Knowledge/Concept/Benchmark`, `/Knowledge/SoftwareApplication/ProductivityTool`.
These structure nodes are already created and should not be recreated. You will only create or update nodes and relations based on the dialogue segment provided.
**Input You Will Receive:**
A JSON object with: `segment_start_timestamp`, `user_turn_text`, `assistant_turn_text`. Assume `user_id` is implicitly "/User" and `assistant_id` is implicitly "/Assistant" for path construction.
**Goal:**
- DO NOT lose ANY information from the dialogue segment (including reasoning).
- DO NOT create nodes or relations that are not explicitly mentioned in the dialogue.
**Output Requirements:**
Your output **MUST** be a single JSON object containing top-level keys: `"add_node"`, `"update_node"`, and `"add_relation"`. Each key will hold a list of operation objects of that type. If no operations of a certain type are needed, provide an empty list `[]` for that key.
**KG Operation Schemas (for items within the lists):**
1. **For items in the `add_node` list:**
`node_details` is an object. It **MUST** contain:
* `"label": "string_primary_human_readable_label_for_node"`
It **MAY** also contain:
* `"description": "string_optional_description_of_the_node_based_on_dialogue_OR_descriptive_summary_for_event_node"`
* `"timestamp": "string_optional_segment_start_timestamp"
Note: The `timestamp` is optional and can be omitted if not relevant. **Example usecase: you can inference the date of the event(not `segment_start_timestamp` but information based on the dialogue context)**.
Example `node_details` for an `add_node` operation:
`{"label": "Todoist", "description": "A task manager."}`
Full operation structure:
```json
{
"operation_type": "add_node", // This key is implicit from the list it's in
"node_id": "/full/canonical/path/to/node",
"node_details": { /* as described above */ }
}
```
2. **For items in the `update_node` list:**
`node_details` is an object containing *only* the attributes to be added or changed, as key-value pairs.
Example `node_details` for an `update_node` operation:
`{"description": "An updated description.", "status": "revised"}`
Full operation structure:
```json
{
"operation_type": "update_node", // Implicit
"node_id": "/full/canonical/path/to/node",
"node_details": { /* as described above */ }
}
```
3. **For items in the `add_relation` list:**
`relation_details` is an object. It **MUST** contain:
* `"label": "string_relationship_label_e.g_INITIATED_BY_HAS_TOPIC"`
Full operation structure:
```json
{
"operation_type": "add_relation", // Implicit
"source_node_id": "/full/path/to/source/node",
"target_node_id": "/full/path/to/target/node",
"relation_details": { /* as described above */ }
}
```
Example"
```json
{
"source_node_id": "/User/RelatedEvents",
"target_node_id": "/Events/LifeExperience/HavingLaunch",
"relation_details": {
"label": "INITIATED"
}
}
```
**Logic Guidance & Node ID Construction:**
1. **Structural Nodes:** Ensure all necessary parent nodes exist using `add_node` operations first (e.g., `/User`, `/User/Profile`, `/Events/[Topic]`, `/Knowledge/[EntityType]/[EntityCategory]`).
2. **Knowledge Entity Node IDs (`/Knowledge/...`):** Identify entity `name`, `EntityType`, and `EntityCategory`. Construct `node_id` as: `/Knowledge/[EntityType]/[EntityCategory]/[SanitizedEntityName]`. Use `add_node`.
3. **Event Node IDs (`/Events/...`):** Classify into a `[Topic]`. Construct `node_id` as: `/Events/[ChosenTopic]/[EventBriefSummary_segment_start_timestamp]` (event end-node format: EventSummary_YYYYMMDD). Use `add_node`, then `add_relation` for links.
4. **User Profile & Status Updates:** Use `update_node` (or `add_node` for sub-nodes if defining them for the first time in this segment) for `/User/Profile` (stable info) and `/User/Status` (temporal info). Be "lazy" with Profile.
5. **Assistant Status Updates:** Typically `update_node` for `/Assistant/Status/CurrentInteractionContext`.
6. **Attributes in `node_details` (for `add_node`):** For the Event Node, the `description` should be a concise summary of the event based on the dialogue. For other nodes, use relevent information that are extracted from context.
7. **Attributes in `node_details` (for `update_node`):** ONLY include attributes being changed or added.
8. **Relation Labels:** Use descriptive labels for `relation_details.label`.
9. **Relation Direction:** Prefer source from event node, except `/User/RelatedEvents` and `/Assistant/RelatedEvents` which should be the source for relations to events.
10. **Relation Preference:** You should NEVER connect to Root node like `/User` or `/Assistant` directly. Instead, connect them to specific sub-nodes like `/User/RelatedEvents` or `/Assistant/RelatedEvents`.
**Constraint:** If information is missing for a required attribute (like `label` or `type` for `add_node`), use a sensible default (e.g., "UnknownType", or derive label from last path segment). Your output must not contain any code comments or additional text outside the main JSON structure.
Appendix
Code: KG Operations Structure
from pydantic import BaseModel
from typing import List, Optional
# --- Pydantic Models for Structured Output ---
class NodeDetails(BaseModel):
label: str
timestamp: Optional[str] = None
description: Optional[str] = None
class AddNodeOperation(BaseModel):
node_id: str
node_details: NodeDetails
class UpdateNodeOperation(BaseModel):
node_id: str
node_details: NodeDetails
class RelationDetails(BaseModel):
label: str
class AddRelationOperation(BaseModel):
source_node_id: str
target_node_id: str
relation_details: RelationDetails
# The endpoint
class KgOperationsResponse(BaseModel):
add_node: Optional[List[AddNodeOperation]]
update_node: Optional[List[UpdateNodeOperation]]
add_relation: Optional[List[AddRelationOperation]]