SysEx messages all mapping to ctrl 85

[Linzmeister]

Hi,

I found a link to the AZ controller on the cakewalk forum today.  WOW!!. 

I have a Yamaha 01V (first version) which used to work as a controller for Sonar up until about Version 6 or 7 when they broke something and 01V control surface plugin I was using (Yamaha 01v Control Surface Module for Sonar Copyright © 2002 Chris Boucher) stopped working.  I was considering buying a controller until I saw this....

I have done some reading on the site and tried to follow the you tube video, but every fader, switch and encoder I touch is saying it is already assigned to the first thing I configured (Main Volume). 

I have set the 01V up in Local Off Mode - which means it outputs only SYSEX data. The reason for running Local off is that my monitoring audio is passing through the sp/dif channels 13/14 and master output.   If I was to stop routing my audio through the 01V, and Turn Local On I saw that the Fader and PAN MIDI messages output are common control change messages, but EQ, dynamics (parameter Wheel) and Aux levels are all SYSEX.

I am running Midi Ox just to see what is coming out raw - I have pruned some lines for clarity:

With Local On
 TIMESTAMP IN PORT STATUS DATA1 DATA2 CHAN NOTE EVENT               
Master Fader
 0004F0C3   1  --     B0    1B    01    1  ---  Control Change       
 0004F0FD   1  --     B0    1B    05    1  ---  Control Change       
 0004F137   1  --     B0    1B    0A    1  ---  Control Change     

Ch15/16
 0004F33A   1  --     B0    0E    03    1  ---  Control Change       
 0004F377   1  --     B0    0E    0A    1  ---  Control Change       
 0004F3AF   1  --     B0    0E    0D    1  ---  Control Change       

ch 13/14
 0004F508   1  --     B0    0D    04    1  ---  Control Change       
 0004F542   1  --     B0    0D    09    1  ---  Control Change       
 0004F57D   1  --     B0    0D    0E    1  ---  Control Change             

Ch12
 0004F6D3   1  --     B0    0C    06    1  ---  Control Change       
 0004F711   1  --     B0    0C    0E    1  ---  Control Change       
 0004F74B   1  --     B0    0C    0F    1  ---  Control Change       

ch 11
 0004F8A1   1  --     B0    0B    07    1  ---  CC: Expression       
 0004F8DE   1  --     B0    0B    0B    1  ---  CC: Expression       
 0004F916   1  --     B0    0B    0C    1  ---  CC: Expression       

Ch10
 0004FA6C   1  --     B0    0A    01    1  ---  CC: PAN               
 0004FAA8   1  --     B0    0A    02    1  ---  CC: PAN               
 0004FB8E   1  --     B0    0A    09    1  ---  CC: PAN                     


with LOCAL OFF
 TIMESTAMP IN PORT STATUS DATA1 DATA2 CHAN NOTE EVENT               

The following line seems to be on a timer of some sort.. keeps appearing when nothing is being touched
 006B992F   1  --     F0  Buffer:     8 Bytes   System Exclusive     
SYSX: F0 43 30 3E 04 25 02 F7

Master Fader
 SYSX: F0 43 10 3E 04 30 00 24 02 F7
 SYSX: F0 43 10 3E 04 30 00 24 0D F7
 SYSX: F0 43 10 3E 04 30 00 24 0E F7

Ch 15/16 fader
 SYSX: F0 43 10 3E 04 30 00 19 03 F7
 SYSX: F0 43 10 3E 04 30 00 19 0A F7

CH 13/14 fader
 SYSX: F0 43 10 3E 04 30 00 18 03 F7
 SYSX: F0 43 10 3E 04 30 00 18 08 F7

SYSX: F0 43 30 3E 04 25 02 F7             Again ??

Ch12 fader
 SYSX: F0 43 10 3E 04 30 00 17 05 F7
 SYSX: F0 43 10 3E 04 30 00 17 0A F7

Ch11 fader
 SYSX: F0 43 10 3E 04 30 00 16 0A F7
 SYSX: F0 43 10 3E 04 30 00 16 0C F7

Ch10 fader
 SYSX: F0 43 10 3E 04 30 00 15 0D F7
 SYSX: F0 43 10 3E 04 30 00 15 12 F7

  SYSX: F0 43 30 3E 04 25 02 F7              Again ??

All of the faders show up on the AZ Last Midi Event as
<43> 10 3e 4 61  0 10 2

I did read where this part was explained indicating already assigned...
   | Ctrl 85

I can follow that as far as
<43> 10 3e 4

but the last part doesn't seem to match up right.
61  0 10 2    | Ctrl 85

Attached is the Startup preset with the my master fader assigned..  I can't assign anything else, because when I select S1 Fader for example, and move fader 1 on the 01V the Assign MIDI button is greyed out because that SYSEX message it is already assigned fro CTRL 85.

It certainly doesn't flow as quickly as the MPK mini example in the youtube video... which I did notice is using control Changes

I am comfortable with programming and very technical details usually... Have I missed some crucial piece of information? 

Regards,
Linz.

[azslow3]

Hi and welcome,

Up to now there was no requests to support parametric SysEx messages. And so they are not supported... yet.

I think "<43> 10 3e 4 61  0 10 2" in fact was sent by Yamaha at some point, it is just not updated then (which is a bug I have to fix). Till now I was using SysEx primary for output, so that functionality was not tested intensively (it looks like I update Last MIDI Event only in case SysEx is different in first 4 bytes).

If you can find any documentation for these SysExes, that can help. Repeatable "F0 43 30 3E 04 25 02 F7" can be "ping" on which some reply is expected. I see there are complicated rules exists (https://www.bome.com/forums/viewtopic.php?t=3734) , so I will probably have to implement Yamaha specific MIDI interpreter. And that can be challenging without known all rules.

In any case, I will fix Last MIDI event bug and I will think about SysEx processing possibilities. Please be patient.

[Linzmeister]

g'day,

Yeah, the last 4 bytes contain the desired (variable) information..

The various different ways I tried to attach pages of the user manual that talk about the MIDI implementation seem to violate your posting rules..  primarily file size.  so a link will have to do.  See Pages 274-289
http://download.yamaha.com/api/asset/file?language=en&site=countrysite-master.prod.wsys.yamaha.com&asset_id=4146  If this is insufficient, I can try and capture the data required off the cable.

I have done a little work with Bome MIDI translator and MIDI OX/MIDI Yoke in the past.  I will try to translate the messages like mentioned on the Bome website and pass the edited messages onto AZ Control Surface.  I have asked @artsrecording if he got it working.  I have known him for 10 years..

After doing some more reading on this forum today and based on symptoms I have I suspect my ACT/XML files are at least partially corrupted or worse.  I will try the ActFix tonight.

So much really useful information here - Thank you.

Regards,
Lindsay.

[azslow3]

Hi,

Thank you for the link. I probably have to dig it deeper. I have already understood that "Local off" mode is for remote controlling another 01V. I still have not understood either the communication is described in this manual. On page 281 there is "4. PARAMETER CHANGE" format. Your observation for Master fader is "F0 43 10 3E 04 30 00 24 02 F7". But in the documentation there is no "type 30". And so I am a bit confused.

I have fixed/extended AZ Controller to make required communication possible. Please download the latest TEST version from Downloads menu, so you need 0.5r1b325 (or later if I find some bug before you download it...).

I attach simple preset which (with some luck) should control current track Volume by Master fader. Can you check that is really the case?
I will extent the preset then to test other features. I think most important are simultaneous moving of several faders and moving faders (I mean feedback). If we can make that work, the rest is much simpler (since not hardware dependent).

[Linzmeister]

Thank you!!

Yup, that works a treat.

Click on any channel in Sonar with the mouse and move master fader on 01V and Sonar's selected fader responds as anticipated.

I have captured a lot of MIDI data... Still making it presentable.  But a little now to show this pattern:

Channel 1
 SYSX: F0 43 10 3E 04 30 00 0C 01 F7
 SYSX: F0 43 10 3E 04 30 00 0C 04 F7
 SYSX: F0 43 10 3E 04 30 00 0C 07 F7

Channel 2
 SYSX: F0 43 10 3E 04 30 00 0D 02 F7
 SYSX: F0 43 10 3E 04 30 00 0D 03 F7
 SYSX: F0 43 10 3E 04 30 00 0D 06 F7

Channel 3
 SYSX: F0 43 10 3E 04 30 00 0E 01 F7
 SYSX: F0 43 10 3E 04 30 00 0E 03 F7
 SYSX: F0 43 10 3E 04 30 00 0E 04 F7

Master Fader
 SYSX: F0 43 10 3E 04 30 00 24 2B F7
 SYSX: F0 43 10 3E 04 30 00 24 2C F7
 SYSX: F0 43 10 3E 04 30 00 24 2D F7


It appears that the third last byte (counting the EOX) is the fader address and the second last byte is the current position value.
From the data I've captured the position filled by the 30 00 does change, but I haven't seen the systematic pattern behind it yet.  More capturing required to see why.

ArtsRecording was absolutely correct about the adjacent faders on the BOME forum:...  I move a single fader and get a 10 byte message, I moved a bunch all at once (with a ruler - which is never going to happen while mixing) and I got an 18 byte message

27 is ch 1 Aux send 1
28 is Ch 2 Aux send 1
....
2F
30 is Ch10 Aux 1...

 SYSX: F0 43 10 3E 04 30 00 30 02 F7
 SYSX: F0 43 30 3E 04 25 02 F7
 SYSX: F0 43 10 3E 04 30 00 28 09 09 08 F7
 SYSX: F0 43 10 3E 04 30 00 2C 06 06 06 05 03 F7
 SYSX: F0 43 10 3E 04 30 00 2B 09 07 07 F7
 SYSX: F0 43 10 3E 04 30 00 2F 06 05 01 F7
 SYSX: F0 43 10 3E 04 30 00 2A 09 F7
 SYSX: F0 43 10 3E 04 30 00 2C 08 08 07 07 06 02 F7
 SYSX: F0 43 10 3E 04 30 00 27 05 0A 0A 0A 0A 09 09 09 09 F7
 SYSX: F0 43 10 3E 04 30 00 30 07 03 F7
 SYSX: F0 43 10 3E 04 30 00 27 06 0B 0B 0B 0B 0A 0A 0A 0A F7
 SYSX: F0 43 10 3E 04 30 00 30 09 05 F7
 SYSX: F0 43 10 3E 04 30 00 27 07 0C 0D 0D 0D 0B 0B 0B 0B F7

As it is 2 AM here, I'd better call it a night.
Thank you once again. 
I'm Really excited to see how far this can go.

[azslow3]

Please slow down a bit with different controls, otherwise I can mix some numbers... Lets agree that you are NOT switching default layout mode, I mean let keep faders controlling channels. We are not done with that yet...

I attach next test preset. This time 15 faders (inclusive master) should control 15 tracks volumes in WAI (WAI are indicated, we have not defined buttes yet on Yama, but you can move WAI with mouse for tests). That should work by one or many (also with a ruler), also test not adjusted faders simultaneously.

If that works, the next topic will be feedback (move faders from movements in Sonar). We will have to find correct messages for that (with some luck the same message sent back, but can be not so simple). Also Yama faders are not touch sensitive and that is going to be tricky. We do not want faders are fighting with your fingers, but we also do not want they stay unsynced. How to achieve that is device specific, we will have to test several approaches.

But first things first, please test attached preset.

[Linzmeister]

Very Nice.

All 15 track faders respond to the 01V fader movements.  I used the spine of a large book to move all 15 01V faders at once and they all moved together.

I made 46 audio tracks in a CWP.  Moving WAI around with the mouse..  01V fader 1 moved WAI first track fader and sequentially up to the master.  After setting some random fader levels, changing WAI region and moving 01V faders a very little, the newly controlled WAI fader jumps to 01V fader level.  That looks perfect!!

The way the old CS plugin I had solved the non-touch sensitive issue was with a Button on the Transport bank that switched MIDI transmission of Automation data on/off.  From memory the Sonar faders still moved on screen, but the 01V was not being sent the automation commands..


Snippet from the old help file:
ON Buttons
The input channels ON buttons control different things, depending on the state of the Return 1 and Return 2 ON buttons:

Return 1    Return 2      ON Button Functions
OFF           OFF                 Mute input or enable aux send, depending on the Fader Mode 
ON            OFF                 Arm Tracks 
OFF           ON                   Transport Controls 

Transport control list
Channel    Function
1                 Return To Zero
2                 Go to End 
3                 Stop 
4                 Play 
5                 Record 
6                 Record Automation 
7                 Previous Measure 
8                 Next Measure 
9                 Previous Marker 
10               Next Marker 
11 - 13/14  Unused 
15/16         On: faders move in response to Sonar
                   Off: ignore fader movements sent by Sonar

The 01V Yahoo Group files area, mentioned in the CW thread, has much more detail about the MIDI implementation than the user manual.  I have only skimmed it so far, but the 30 00 xx may be a 3 byte address or a function call with a parameter followed by an address.  Surprisingly this beasty will also transmit and receive meter levels.

[azslow3]

So far so good...

I have tried to dig "community" documentation, unfortunately it looks like messages we observe are from "proprietary" part (the chapter exists in the documentation, but it is deleted). So, lets continue with what we have and look how far we can get with that.

General Motor Off is not a problem, we will implement it later (as in the old plug-in or some other way).

Next test is to find good way to send values from Sonar to fader while not fighting with finger when you move it manually. Sound like a no problem, but you will see it practice soon

For attached preset please downloaded new test version, b329. I have fixed some bugs which by luck I hit when preparing that preset, it took a while to understand from where they come, so it is probably late now for you again...

You will need a project with at least 3 tracks. The test is for the first 3 faders only.

* After loading the preset, switch to the "Overview" tab. In the "Current Software State" you should see "Moved(_Ch) (No)". Select "Motor (Off)" there. Then change "Off" to "On" in the right box (so you see "Motor (On)").
* 3 faders should move to current position in Sonar. If that does not happened, the message is incorrect (not the same as from Yama). Let me know that is the case, ignore other questions in this post then since they make no sense
* if you observe something strange from Yama, for example some fader is "busy moving", switch "Motor" to "Off". Do not let it "vibrate" for too long, that can damage the device! Let me know which fader it was (all 3 or just one of them) and do not follow future questions.
* if faders was moved correctly, try to change volume of first WAI track in Sonar. Does Yama fader moves as expected? Some strange movements/sounds?  "Busy" movement started?
* thy move second and then third volumes in Sonar, the same questions as for the first

* try change volume with these 3 faders separately (I mean moving hardware faders). Does at least one of them react correctly? Are they fighting with fingers? Are they "jumping" (even a bit) after you stop moving them?


Some explanation. Each of these 3 faders is configured with different logic when should Sonar send current value back. It can happened that yet another method is required. But it can happened that some (or even all) work correctly.

[Linzmeister]

Hi,

All faders move to WAI fader values when I turned Motors on in the plug in.
There was no motor noise by simply enabling motor data transmission after initial re-positioning to current WAI value.
Moving first 3 faders in WAI region moves first 3 faders on 01V.

Move 01V fader 1:              The motor was fighting me for control.

Move 01V Faders 2 and 3:  Both moved as expected and both felt natural and the motors were not fighting me for control.

I then Record Armed tracks 2 and 3 and enabled write automation on 2&3.  recorded a bunch of moves and it played back as expected.

It has been a long time since I have seen it do that.  Very satisfying to watch.. 

Thank you.

[azslow3]

That is better then I have thought 

But we should decide which method to use, 2 or 3. I attach test 4 preset.
First 7 faders there are (should be) of type "2" and next 7 faders are of type "3".

How feedback cycle is working, when you move hardware fader:
* AZC ask Sonar to change strip volume. Sonar changes the volume (if that is possible)
* Next time (~75 ms) AZC "notice" the change in volume and then...

The difference:
* type 2 (1-7) "skip" one (and only one) feedback from Sonar. With the idea that since the change was triggered by hardware fader, we do not have to send it back. But all other changes will be transmitted to Y then.
* type 3 (8-14) stop transmitting updates to Y for ~5 seconds (that we can change to 1-2 sec, I have specially made that time big for testing). But then it ALWAYS transmit actual position, either it was the result of original movement or from other operations Sonar.

Both methods have advantages and potential disadvantages.
Possible deal breaker is a disadvantage for type 3 (8-14), so please check either that is the case: since type 3 ALWAYS send actual position after 5 seconds without operation, if Y is not "smart", it will activate motor even if fader position is ok, so it will "tick moved".
How to test: move some fader of type 3 (8-14), then stop moving but keep your finger on it. Do you feel something after ~5 seconds? It can be also tested visually, but the effect can be really subtle.

You can ask why I would prefer (3) over (2), in case Y is sufficiently smart to NOT move faders when not required. Disadvantage of (2) is that in case you have already recorded automation and try to move the fader, Y will fight with you all the time. (3) will stop updating for a while.

We can try some "hybrid" solution if we find that (3) is better but mentioned disadvantage exists. But at the end that is you who can decide what you prefer, so please test and let me know what you think.

Note. It can happened that some faders are not moved to right position initially. SysExs are slow in transmission, so when 14 of them are sent simultaneously, some can be "skipped"  on Y side. That is why Y use "cumulative" messages when you adjust several faders. I will adopt that approach once we decide about feedback type, so I will send only 2 SysEx to update 14 faders.

[Linzmeister]

I could not hear, feel or see any motor ticks on either bank of type 2 or 3 after being moved.

That is good.

So I recorded automation on all 14 tracks. 
I moved NOW time to the middle of the automation sequence.  Within each bank of 7 tracks, some had solid envelope trace between 2 nodes(currently changing value), some had dotted line envelopes between 2 nodes (static resting automation).

Sonar has different behavior with automations, which is a combination of parameter change mode used in the Control Surface (so set in my Preset) and Track Automation mode (set in the Track Inspector, Track tab, Automation section).
Parameter change modes are:
* Timeout (I use it now). When you change parameter, the Automation is written during the time you change parameter and some time after. Once that "timeout" (Sonar internal, I do not know how to change it) is over, the automation is no longer indicated as active. The result depends from Track Automation mode
* Automatic. Like Timeout, but was thought to be "smart". So Sonar tries to deduct either you really still want write automation or not. It normally fails in that, producing rather strange effects. For example you have moved fader to -6dB and kept it here for a while, than moved to -3dB. If you do thick "quick", so the values transmitted during -6...-3 movement are not -6,-5,-4,-3 but simply new change, f.e. -4,-3, the line is drawn from -6 to -4. So, during the time when your fader was not moved on -6, the automation is written as slowly moving from -6 to -4. So I prefer not use this mode (but is is supported in plug-in).
* Manual. When explicit signal is sent from Surface when to start and when end automation writing. That is obviously the best for overwriting automations, since no guesses should be made either you want the value stay as you want or you expect it is reverted to existing automation. You kind of "punch in" manually specifying the region. Naturally that works with touch sensitive controls, so you overwrite as long as you touch it. Yama has no touch sensitive faders... but we can try to use it later, using some button as a "touch sensor". This mode is used with mouse, since mouse press/release can be used as "touch".

Track automation modes are:
* Touch (default). Obviously works best with Manual Parameter Set mode. The automation is (over)written only as long and Set mode say it should be
* Overwrite. The automation is overwritten to current value. So, "no longer write" signal is irrelevant.
* Latch. The same as Overwrite, but it start (over)writing not from the beginning, but from the first time you change it.

For not touch sensitive faders, Latch/Overwrite are more natural then Touch.

"Doted" parts is when the Timeout was over. And since you was in default "Touch" mode, no automation is written there.

With the transport stopped, the tracks with solid automation envelope lines behaved weird..     
...
Is this a normal sonar/recorded automation symptom?

Yes. In fact when Automation reading is enabled, your real level is the automation. As soon as you release fader, the value is reverted to automated.

So, Sonar Fader is not in sync with real volume! And mode (3) fader in that particular case also (which is not good and I have to fix it).

Thank you.  This is progressing so much faster than I thought possible.

Not today... quite some activity there. Not sure I can prepare next next till tomorrow.

[Linzmeister]

Excellent explanation.
I had to read it twice to fully understand the finer implications of the various modes and how they impact the operation and workflow.

You're right, I had not deliberately chosen any particular automation mode so, yes it was set to Touch.

[azslow3]

Next Test  You need b330 test version (or later) for it.

Faders 1-15 should control 15 WAI Tracks volumes.
Programmed method 3 (from previous discussion) for all faders.
Feedback time (delay till faders start follow Sonar position after you have used it) is set to 2 Seconds.
One note about that which I have not mentioned before: if you move several faders, simultaneously or sequestial within 2 sec, feedback for all of them is restored 2 sec after you no longer operate ANY fader (that, as well as everything else can be changed. But that is how it is configured now).
Faders should "jump back" after moving fader in Sonar, there is an automation for it and it is enabled (R button). So that will put hardware fader out of sync with Sonar faders, but keep them in sync with real volume value.
Data are transmitted back "compressed" way, up to 7 channels in one SysEx. With the same logic as Yama send changes.

The whole preset was reworked to implement that logic, so there can be some general glitches (as you understand, I have hard time testing it, I always have to change SysEx sends and loop them back, so I can easily forget to revert the changes  before publishing the preset) . Do not worry, just report what I have managed to break.

Otherwise check all possible operations with faders, this preset supposed to be "final" for fader communication logic. I hope we can move to buttons and encoders soon.

[Linzmeister]

G'day Alexey,

Is there any particular Automation mode you want me testing in or do you want me to test all 3 modes? Touch, Overwrite, Latch?

[azslow3]

I want to check that faders are working correctly in all situations. Better with just uploaded b332 (there is some tiny modification which affect mapping from faders).

In other words we should check that I have not overseen something in configuration, so faders are moved when required and as expected (also all of them, initially and in case of WAI changes). If you look inside the preset, there is quite some "programming", and as with every program, bugs are easer to fix during the development and not months later when the program looks like was written by aliens