Skip to content

Commit 0cd409d

Browse files
kefniarkkefniark
committed
Add FAQ and playground page
1 parent c378f9b commit 0cd409d

File tree

11 files changed

+1222
-0
lines changed

11 files changed

+1222
-0
lines changed

‎.github/workflows/playground.yml

+50
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Simple workflow for deploying static content to GitHub Pages
2+
name: Deploy static content to Pages
3+
4+
on:
5+
# Runs on pushes targeting the default branch
6+
push:
7+
branches: ["develop"]
8+
9+
# Allows you to run this workflow manually from the Actions tab
10+
workflow_dispatch:
11+
12+
# Sets permissions of the GITHUB_TOKEN to allow deployment to GitHub Pages
13+
permissions:
14+
contents: read
15+
pages: write
16+
id-token: write
17+
18+
# Allow one concurrent deployment
19+
concurrency:
20+
group: "pages"
21+
cancel-in-progress: true
22+
23+
jobs:
24+
# Single deploy job since we're just deploying
25+
deploy:
26+
environment:
27+
name: github-pages
28+
url: ${{ steps.deployment.outputs.page_url }}
29+
runs-on: ubuntu-latest
30+
steps:
31+
- name: Checkout
32+
uses: actions/checkout@v3
33+
- uses: actions/setup-node@v3
34+
with:
35+
node-version: 18
36+
cache: 'npm'
37+
- run: |
38+
cd playground
39+
yarn
40+
yarn build
41+
- name: Setup Pages
42+
uses: actions/configure-pages@v2
43+
- name: Upload artifact
44+
uses: actions/upload-pages-artifact@v1
45+
with:
46+
# Upload entire repository
47+
path: './playground/dist'
48+
- name: Deploy to GitHub Pages
49+
id: deployment
50+
uses: actions/deploy-pages@v1

‎docs/faq.md

+80
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
# Frequently Asked Question
2+
3+
* [Language Detection Error](#my-text-is-detected-in-the-wrong-language)
4+
* [Cand I have a custom version](#can-i-have-a-version-specific-for-my-app-and-my-needs)
5+
* [Short text detection issues](#can-tinyld-identify-short-strings)
6+
* [Live Chat usage](#can-i-use-tinyld-for-an-application-like-a-chat-even-if-texts-are-short)
7+
8+
---
9+
10+
## My text is detected in the wrong language
11+
12+
It's sad to hear, but it's not unusual.
13+
14+
As we can see [here](https://github.com/komodojp/tinyld/blob/develop/docs/benchmark.md#libraries), **Tinyld** is good but not perfect. Overall 1~2% of the time it will get it wrong.
15+
16+
The two things which usually increase error rate:
17+
* short inputs, try to make it longer
18+
* similar language (like spanish and catalan)
19+
* generic names/brand which may appears in multiple language corpus
20+
21+
---
22+
23+
## Can I have a version specific for my app and my needs
24+
25+
Everything in life is about tradeoff.
26+
27+
Tinyld was designed to be accurate, small and fast.
28+
Based on how much space and resource you are ready to spend, we provide different flavor
29+
30+
- **Tinyld** : The general one (~500KB) which detect 64 languages
31+
- **Tinyld Light** : Mostly for browser usage (~70KB) which detect 24 languages
32+
- **Tinyld Heavy** : The one for backend usage (few MB) which focus on accuracy only
33+
34+
To select the one you want, simply change your import
35+
```ts
36+
import { detect } from 'tinyld'
37+
import { detect } from 'tinyld/light'
38+
import { detect } from 'tinyld/heavy'
39+
```
40+
41+
---
42+
43+
## Can tinyld identify short strings?
44+
45+
If by short you mean one or two word with a good accuracy, the answer is most likely **No**.
46+
47+
The key point here is to understand the algorithms behind language detection.
48+
* How can you detect a text language, from many possibilities without embedding a dictionary of each language?
49+
* Even just between 2 or 3 languages, how would you do it? Handcraft regexp for specific languages?
50+
* How can you scale up this method easily to more languages?
51+
52+
There are multiple approaches to solve this problem, but the two main ones are AI and statistics.
53+
And the general idea is to recognize some patterns or succession of letters that are specific to each language. ([n-gram](https://en.wikipedia.org/wiki/N-gram))
54+
55+
**Good part**:
56+
* we don't need to understand a language syntax to detect it
57+
* can be extended to more language fairly easily
58+
* the signature of a language can be quite small only few KB
59+
60+
**Bad part**:
61+
* it require a certain size of text to get a good detection level and valuable n-grams
62+
* mixed language content is hard to detect
63+
64+
We are always looking for way to improve our process, and you can find some benchmark [related to this](https://github.com/komodojp/tinyld/blob/develop/docs/benchmark.md#accuracy-by-text-length).
65+
But to give some numbers:
66+
* Tinyld usually pass the ~95% detection accuracy threshold around ~24 characters
67+
* It fall at ~80% for 12 characters (barely usable)
68+
* Less than 10 characters it's just random
69+
70+
---
71+
72+
## Can I use tinyld for an application like a chat, even if texts are short?
73+
74+
Yes you can, and this is why it was built originally.
75+
76+
One of the easy ways to workaround the size issue is to keep a context, a user is unlikely to change language abruptly in the middle of a discussion. And multiple users usually chat in a common language.
77+
So you can keep some buffer (like the last 256 characters of this user in this channel) and check this and not just the last message.
78+
79+
This gives stability and more accurate results to the detection.
80+

‎playground/.postcssrc.cjs

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
module.exports = {
2+
plugins: {
3+
tailwindcss: {}
4+
}
5+
}

‎playground/index.html

+13
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
<!DOCTYPE html>
2+
<html lang="en">
3+
<head>
4+
<meta charset="UTF-8" />
5+
<link rel="icon" type="image/svg+xml" href="/vite.svg" />
6+
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
7+
<title>Tinyld Playground</title>
8+
</head>
9+
<body>
10+
<div id="app"></div>
11+
<script type="module" src="/src/main.js"></script>
12+
</body>
13+
</html>

‎playground/package.json

+27
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
{
2+
"name": "tinyld-playground",
3+
"private": true,
4+
"version": "0.0.0",
5+
"type": "module",
6+
"scripts": {
7+
"dev": "vite",
8+
"build": "vite build",
9+
"preview": "vite preview"
10+
},
11+
"dependencies": {
12+
"@fortawesome/fontawesome-svg-core": "^6.2.0",
13+
"@fortawesome/free-brands-svg-icons": "^6.2.0",
14+
"@fortawesome/free-regular-svg-icons": "^6.2.0",
15+
"@fortawesome/free-solid-svg-icons": "^6.2.0",
16+
"@fortawesome/vue-fontawesome": "^3.0.2",
17+
"@vueuse/core": "^9.5.0",
18+
"tailwindcss": "^3.2.3",
19+
"tinyld": "^1.3.1",
20+
"vue": "^3.2.41"
21+
},
22+
"devDependencies": {
23+
"@vitejs/plugin-vue": "^3.2.0",
24+
"postcss": "^8.4.12",
25+
"vite": "^3.2.3"
26+
}
27+
}

‎playground/src/App.vue

+210
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
<template>
2+
<div>
3+
<div class="area" >
4+
<div class="absolute flex w-full h-full justify-center items-center">
5+
<div class="w-[600px] max-w-[80vw] font-bold text-4xl z-10">
6+
<h1 class="text-slate-50">TinyLD Playground</h1>
7+
<p class="text-sm font-normal mt-2 text-slate-300">
8+
Tiny Language Detector, simply detect the language of a unicode UTF-8 text
9+
</p>
10+
<textarea
11+
rows="8"
12+
v-model="input"
13+
placeholder="Enter your text here"
14+
class="block w-full px-3 py-2 bg-white border border-slate-300 rounded-md text-sm shadow-sm placeholder-slate-400
15+
focus:outline-none focus:border-sky-500 focus:ring-1 focus:ring-sky-500 font-normal mt-5"/>
16+
17+
<div v-if="results && results.length > 0" class="font-normal bg-white border shadow-sm rounded-md mt-2 px-3 py-2 text-sm">
18+
<ul class="list-item">
19+
<li v-for="(result, index) in results" :key="result.lang" class="flex items-center pt-1">
20+
<span style="width: 80px" class="px-2 font-semibold uppercase">
21+
{{ result.lang }}
22+
</span>
23+
<div class="w-full bg-gray-100 rounded-full h-2.5 dark:bg-gray-700">
24+
<div v-if="index === 0" class="bg-blue-400 h-2.5 rounded-full" :style="`width: ${Math.round(result.accuracy * 10000)/100 }%`"></div>
25+
<div v-else class="bg-yellow-200 h-2.5 rounded-full" :style="`width: ${Math.round(result.accuracy * 10000)/100 }%`"></div>
26+
</div>
27+
<span style="width: 80px" class="px-2">
28+
{{ Math.round(result.accuracy * 10000)/100 }}%
29+
</span>
30+
</li>
31+
</ul>
32+
</div>
33+
34+
<div class="flex gap-3 pt-4">
35+
<a href="https://github.com/komodojp/tinyld" alt="Github" target="_blank" class="text-slate-200 hover:text-slate-50">
36+
<v-icon label="github" :icon="{ prefix: 'fab', iconName: 'github' }" />
37+
</a>
38+
<a href="https://github.com/komodojp/tinyld/blob/develop/docs/benchmark.md" alt="Benchmark" target="_blank" class="text-slate-200 hover:text-slate-50">
39+
<v-icon label="benchmark" icon="chart-line" />
40+
</a>
41+
<a href="https://github.com/komodojp/tinyld/blob/develop/docs/faq.md" alt="FAQ" target="_blank" class="text-slate-200 hover:text-slate-50">
42+
<v-icon label="FAQ" :icon="{ prefix: 'far', iconName: 'circle-question' }" />
43+
</a>
44+
<a href="#" alt="Share" class="text-slate-200 hover:text-slate-50" @click="copy()">
45+
<v-icon label="Share" icon="share-from-square" />
46+
<span v-if="copied" class="text-slate-50 pl-2 text-lg">Copied to clipboard !</span>
47+
</a>
48+
</div>
49+
</div>
50+
</div>
51+
52+
<ul class="circles">
53+
<li></li>
54+
<li></li>
55+
<li></li>
56+
<li></li>
57+
<li></li>
58+
<li></li>
59+
<li></li>
60+
<li></li>
61+
<li></li>
62+
<li></li>
63+
</ul>
64+
</div >
65+
</div>
66+
</template>
67+
68+
<script setup>
69+
import { ref, computed } from 'vue'
70+
import { detectAll } from 'tinyld'
71+
import { useClipboard } from '@vueuse/core'
72+
73+
const input = ref('')
74+
const results = computed(() => {
75+
return detectAll(input.value)
76+
})
77+
const link = computed(() => {
78+
return `https://tinyld.vercel.app/?text=${encodeURIComponent(input.value)}`
79+
})
80+
81+
const clipboard = useClipboard({ source: link })
82+
83+
const copied = ref(false)
84+
const copy = () => {
85+
clipboard.copy()
86+
copied.value = true
87+
setTimeout(() => {
88+
copied.value = false
89+
}, 2000)
90+
}
91+
</script>
92+
93+
<style scoped>
94+
.area{
95+
background: #4e54c8;
96+
background: -webkit-linear-gradient(to left, #8f94fb, #4e54c8);
97+
width: 100%;
98+
height:100vh;
99+
}
100+
101+
.circles{
102+
position: absolute;
103+
top: 0;
104+
left: 0;
105+
width: 100%;
106+
height: 100%;
107+
overflow: hidden;
108+
}
109+
110+
.circles li{
111+
position: absolute;
112+
display: block;
113+
list-style: none;
114+
width: 20px;
115+
height: 20px;
116+
background: rgba(255, 255, 255, 0.2);
117+
animation: animate 25s linear infinite;
118+
bottom: -150px;
119+
120+
}
121+
122+
.circles li:nth-child(1){
123+
left: 25%;
124+
width: 80px;
125+
height: 80px;
126+
animation-delay: 0s;
127+
}
128+
129+
130+
.circles li:nth-child(2){
131+
left: 10%;
132+
width: 20px;
133+
height: 20px;
134+
animation-delay: 2s;
135+
animation-duration: 12s;
136+
}
137+
138+
.circles li:nth-child(3){
139+
left: 70%;
140+
width: 20px;
141+
height: 20px;
142+
animation-delay: 4s;
143+
}
144+
145+
.circles li:nth-child(4){
146+
left: 40%;
147+
width: 60px;
148+
height: 60px;
149+
animation-delay: 0s;
150+
animation-duration: 18s;
151+
}
152+
153+
.circles li:nth-child(5){
154+
left: 65%;
155+
width: 20px;
156+
height: 20px;
157+
animation-delay: 0s;
158+
}
159+
160+
.circles li:nth-child(6){
161+
left: 75%;
162+
width: 110px;
163+
height: 110px;
164+
animation-delay: 3s;
165+
}
166+
167+
.circles li:nth-child(7){
168+
left: 35%;
169+
width: 150px;
170+
height: 150px;
171+
animation-delay: 7s;
172+
}
173+
174+
.circles li:nth-child(8){
175+
left: 50%;
176+
width: 25px;
177+
height: 25px;
178+
animation-delay: 15s;
179+
animation-duration: 45s;
180+
}
181+
182+
.circles li:nth-child(9){
183+
left: 20%;
184+
width: 15px;
185+
height: 15px;
186+
animation-delay: 2s;
187+
animation-duration: 35s;
188+
}
189+
190+
.circles li:nth-child(10){
191+
left: 85%;
192+
width: 150px;
193+
height: 150px;
194+
animation-delay: 0s;
195+
animation-duration: 11s;
196+
}
197+
198+
@keyframes animate {
199+
0%{
200+
transform: translateY(0) rotate(0deg);
201+
opacity: 1;
202+
border-radius: 0;
203+
}
204+
100%{
205+
transform: translateY(-1000px) rotate(720deg);
206+
opacity: 0;
207+
border-radius: 50%;
208+
}
209+
}
210+
</style>

0 commit comments

Comments
 (0)