PMID- 35161879
OWN - NLM
STAT- MEDLINE
DCOM- 20220216
LR  - 20220219
IS  - 1424-8220 (Electronic)
IS  - 1424-8220 (Linking)
VI  - 22
IP  - 3
DP  - 2022 Feb 2
TI  - Beyond the Edge: Markerless Pose Estimation of Speech Articulators from 
      Ultrasound and Camera Images Using DeepLabCut.
LID - 10.3390/s22031133 [doi]
LID - 1133
AB  - Automatic feature extraction from images of speech articulators is currently 
      achieved by detecting edges. Here, we investigate the use of pose estimation deep 
      neural nets with transfer learning to perform markerless estimation of speech 
      articulator keypoints using only a few hundred hand-labelled images as training 
      input. Midsagittal ultrasound images of the tongue, jaw, and hyoid and camera 
      images of the lips were hand-labelled with keypoints, trained using DeepLabCut 
      and evaluated on unseen speakers and systems. Tongue surface contours 
      interpolated from estimated and hand-labelled keypoints produced an average mean 
      sum of distances (MSD) of 0.93, s.d. 0.46 mm, compared with 0.96, s.d. 0.39 mm, 
      for two human labellers, and 2.3, s.d. 1.5 mm, for the best performing edge 
      detection algorithm. A pilot set of simultaneous electromagnetic articulography 
      (EMA) and ultrasound recordings demonstrated partial correlation among three 
      physical sensor positions and the corresponding estimated keypoints and requires 
      further investigation. The accuracy of the estimating lip aperture from a camera 
      video was high, with a mean MSD of 0.70, s.d. 0.56 mm compared with 0.57, s.d. 
      0.48 mm for two human labellers. DeepLabCut was found to be a fast, accurate and 
      fully automatic method of providing unique kinematic data for tongue, hyoid, jaw, 
      and lips.
FAU - Wrench, Alan
AU  - Wrench A
AD  - Clinical Audiology, Speech and Language Research Centre, Queen Margaret 
      University, Musselburgh EH21 6UU, UK.
AD  - Articulate Instruments Ltd., Musselburgh EH21 6UU, UK.
FAU - Balch-Tomes, Jonathan
AU  - Balch-Tomes J
AD  - Articulate Instruments Ltd., Musselburgh EH21 6UU, UK.
LA  - eng
PT  - Journal Article
DEP - 20220202
PL  - Switzerland
TA  - Sensors (Basel)
JT  - Sensors (Basel, Switzerland)
JID - 101204366
SB  - IM
MH  - Biomechanical Phenomena
MH  - *Dental Articulators
MH  - Humans
MH  - Jaw/diagnostic imaging
MH  - Lip/diagnostic imaging
MH  - *Speech
MH  - Tongue/diagnostic imaging
PMC - PMC8838804
OTO - NOTNLM
OT  - keypoints
OT  - landmarks
OT  - lip reading
OT  - multimodal speech
OT  - pose estimation
OT  - speech kinematics
OT  - ultrasound tongue imaging
COIS- The authors declare no conflict of interest.
EDAT- 2022/02/16 06:00
MHDA- 2022/02/17 06:00
PMCR- 2022/02/02
CRDT- 2022/02/15 01:12
PHST- 2021/11/29 00:00 [received]
PHST- 2022/01/25 00:00 [revised]
PHST- 2022/01/28 00:00 [accepted]
PHST- 2022/02/15 01:12 [entrez]
PHST- 2022/02/16 06:00 [pubmed]
PHST- 2022/02/17 06:00 [medline]
PHST- 2022/02/02 00:00 [pmc-release]
AID - s22031133 [pii]
AID - sensors-22-01133 [pii]
AID - 10.3390/s22031133 [doi]
PST - epublish
SO  - Sensors (Basel). 2022 Feb 2;22(3):1133. doi: 10.3390/s22031133.